Uploading an Excel with Data Shuttle Containing Duplicate Rows
I have a Data Upload workflow via Data Shuttle that runs on attachment for the most recent file. The excel files that stakeholders are uploading usually only have ~10 rows and when it goes through the workflow, the rows self-populate into the correct positions within a Smartsheet master database. The problem I have encountered is that sometimes the data being uploaded to Data Shuttle could contain a duplicate row that already exists within the Smartsheet. I want duplicates to be added to the bottom of the sheet like the rest of the new data already is, regardless of whether it is a duplicate or not. When there is a duplicate, Smartsheet doesn’t import that row into the sheet. Please let me know if this makes sense and any suggestions if you have them would be greatly appreciated.
Answers
-
Kleerfyre ✭✭✭✭✭✭Each row will have to have a unique identifier for it to work correctly. Is it possible for you to convert this process into your users using a form so that it will always add an new entry into your sheet?
Jonathan Sanders, CSM
"Change is always scary because it is unknown, but facing the unknown is what makes us stronger."
-
cko ✭
As of right now, our stakeholders use a form to upload the data into the sheet for Data Shuttle to do its thing. We are still in the preliminary phases of planning out this project but given the frequency of data being uploaded and the need for automation, we are really trying to stick to a model where an excel is uploaded (rows being auto populated into the Smartsheet database regardless if they are duplicates or not) and the rest taken care of. Someone from in my company suggested to look into if there is a way to not designate a unique record column but this doesn't seem to be the case. The only other thing I could think of would be to create a new Smartsheet sheet for every data upload but I don't think this is even feasible given the fact that many other sheets are linked to this main sheet via formulas.
-
Kleerfyre ✭✭✭✭✭✭
Why not just cut out the Data Shuttle part? You could use the form for the stakeholders to fill out instead of them typing it into an excel then clicking on a form to upload the excel to the sheet to have data shuttle grab it and then put it on the sheet.
Jonathan Sanders, CSM
"Change is always scary because it is unknown, but facing the unknown is what makes us stronger."
-
cko ✭
I wish I could cut Data Shuttle out of the picture. Unfortunately Data Shuttle has to be used because there is a separate database kept in another system used by a different company that generates an excel file for us to upload into our system.
-
Lesa Weaver ✭✭✭
HELLO@cko,
I had an issue with duplicate unique identifiers that I solved by having a column in my Excel spreadsheet that concatenates two columns to form a unique identifier column that is better for my Smartsheet. You could use a formula combining your current unique identifier + today so that it would always be unique within that day when it is uploaded. If you have duplicates within one day, you could add time.
In Smartsheet, I don't need the unique identifier primary column anymore after upload because I have the data in separate columns already, so I just hide that column.
Hope this helps.
Categories
<\/p>
=VALUE(IFERROR(JOIN(DISTINCT(COLLECT([Customer PO Amount (USD)]@row:[Customer PO Amount (Local Currency)]@row, [Customer PO Amount (USD)]@row:[Customer PO Amount (Local Currency)]@row, <>\"\"))), \" \"))<\/p>"}]}},"status":{"statusID":3,"name":"Accepted","state":"closed","recordType":"discussion","recordSubType":"question"},"bookmarked":false,"unread":false,"category":{"categoryID":321,"name":"Smartsheet Basics","url":"https:\/\/community.smartsheet.com\/categories\/smartsheet-basics%2B","allowedDiscussionTypes":[]},"reactions":[{"tagID":3,"urlcode":"Promote","name":"Promote","class":"Positive","hasReacted":false,"reactionValue":5,"count":0},{"tagID":5,"urlcode":"Insightful","name":"Insightful","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":11,"urlcode":"Up","name":"Vote Up","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":13,"urlcode":"Awesome","name":"Awesome","class":"Positive","hasReacted":false,"reactionValue":1,"count":0}],"tags":[{"tagID":254,"urlcode":"formulas","name":"Formulas"}]},{"discussionID":109460,"type":"question","name":"IF \/ OR formula then Check a Box","excerpt":"I need a formula that IF a specific column has either 1 of 2 specific choices, then a BOX in another column is checked: =IF(([Study Status]@row = \"Active\/ Open to Accrual\", 1) OR([Study Status]@row = \"In Start-Up\", 1))","snippet":"I need a formula that IF a specific column has either 1 of 2 specific choices, then a BOX in another column is checked: =IF(([Study Status]@row = \"Active\/ Open to Accrual\", 1)…","categoryID":321,"dateInserted":"2023-08-25T13:30:39+00:00","dateUpdated":null,"dateLastComment":"2023-08-25T16:21:27+00:00","insertUserID":9250,"insertUser":{"userID":9250,"name":"Susan Swisher","url":"https:\/\/community.smartsheet.com\/profile\/Susan%20Swisher","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/defaultavatar\/nWRMFRX6I99I6.jpg","dateLastActive":"2023-08-25T16:23:49+00:00","banned":0,"punished":0,"private":false,"label":"✭✭✭✭✭"},"updateUserID":null,"lastUserID":161714,"lastUser":{"userID":161714,"name":"Carson Penticuff","url":"https:\/\/community.smartsheet.com\/profile\/Carson%20Penticuff","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/userpics\/B0Q390EZX8XK\/nBGT0U1689CN6.jpg","dateLastActive":"2023-08-26T01:04:51+00:00","banned":0,"punished":0,"private":false,"label":"✭✭✭✭✭✭"},"pinned":false,"pinLocation":null,"closed":false,"sink":false,"countComments":9,"countViews":45,"score":null,"hot":3385956126,"url":"https:\/\/community.smartsheet.com\/discussion\/109460\/if-or-formula-then-check-a-box","canonicalUrl":"https:\/\/community.smartsheet.com\/discussion\/109460\/if-or-formula-then-check-a-box","format":"Rich","tagIDs":[254],"lastPost":{"discussionID":109460,"commentID":392616,"name":"Re: IF \/ OR formula then Check a Box","url":"https:\/\/community.smartsheet.com\/discussion\/comment\/392616#Comment_392616","dateInserted":"2023-08-25T16:21:27+00:00","insertUserID":161714,"insertUser":{"userID":161714,"name":"Carson Penticuff","url":"https:\/\/community.smartsheet.com\/profile\/Carson%20Penticuff","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/userpics\/B0Q390EZX8XK\/nBGT0U1689CN6.jpg","dateLastActive":"2023-08-26T01:04:51+00:00","banned":0,"punished":0,"private":false,"label":"✭✭✭✭✭✭"}},"breadcrumbs":[{"name":"Home","url":"https:\/\/community.smartsheet.com\/"},{"name":"Get Help","url":"https:\/\/community.smartsheet.com\/categories\/get-help"},{"name":"Smartsheet Basics","url":"https:\/\/community.smartsheet.com\/categories\/smartsheet-basics%2B"}],"groupID":null,"statusID":3,"attributes":{"question":{"status":"accepted","dateAccepted":"2023-08-25T15:13:03+00:00","dateAnswered":"2023-08-25T14:14:39+00:00","acceptedAnswers":[{"commentID":392575,"body":"
Give this a try:<\/p>
=IF(OR([Study Status]@row = \"Active\/Open to Accrual\", [Study Status]@row = \"In Start-Up\"), 1)<\/p>"}]}},"status":{"statusID":3,"name":"Accepted","state":"closed","recordType":"discussion","recordSubType":"question"},"bookmarked":false,"unread":false,"category":{"categoryID":321,"name":"Smartsheet Basics","url":"https:\/\/community.smartsheet.com\/categories\/smartsheet-basics%2B","allowedDiscussionTypes":[]},"reactions":[{"tagID":3,"urlcode":"Promote","name":"Promote","class":"Positive","hasReacted":false,"reactionValue":5,"count":0},{"tagID":5,"urlcode":"Insightful","name":"Insightful","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":11,"urlcode":"Up","name":"Vote Up","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":13,"urlcode":"Awesome","name":"Awesome","class":"Positive","hasReacted":false,"reactionValue":1,"count":0}],"tags":[{"tagID":254,"urlcode":"formulas","name":"Formulas"}]},{"discussionID":109457,"type":"question","name":"Conditional Formatting (modified date not in the last 3 weeks)","excerpt":"I would like to use Conditional Formatting (highlight the entire row yellow) if the modified date is not within the last 3 weeks. So any row that has not been recently updated (last 3 weeks) should be highlighted in yellow. Is this spmething that can be done directly within conditional formatting or should I first create a…","snippet":"I would like to use Conditional Formatting (highlight the entire row yellow) if the modified date is not within the last 3 weeks. So any row that has not been recently updated…","categoryID":321,"dateInserted":"2023-08-25T12:33:14+00:00","dateUpdated":null,"dateLastComment":"2023-08-25T12:40:57+00:00","insertUserID":161267,"insertUser":{"userID":161267,"name":"Jef Snyders","title":"Jef Snyders","url":"https:\/\/community.smartsheet.com\/profile\/Jef%20Snyders","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/userpics\/4HJAEW33KBD0\/nXEKEZE5EQEV4.jpg","dateLastActive":"2023-08-25T13:32:31+00:00","banned":0,"punished":0,"private":false,"label":"✭✭"},"updateUserID":null,"lastUserID":161267,"lastUser":{"userID":161267,"name":"Jef Snyders","title":"Jef Snyders","url":"https:\/\/community.smartsheet.com\/profile\/Jef%20Snyders","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/userpics\/4HJAEW33KBD0\/nXEKEZE5EQEV4.jpg","dateLastActive":"2023-08-25T13:32:31+00:00","banned":0,"punished":0,"private":false,"label":"✭✭"},"pinned":false,"pinLocation":null,"closed":false,"sink":false,"countComments":2,"countViews":26,"score":null,"hot":3385935251,"url":"https:\/\/community.smartsheet.com\/discussion\/109457\/conditional-formatting-modified-date-not-in-the-last-3-weeks","canonicalUrl":"https:\/\/community.smartsheet.com\/discussion\/109457\/conditional-formatting-modified-date-not-in-the-last-3-weeks","format":"Rich","tagIDs":[437],"lastPost":{"discussionID":109457,"commentID":392553,"name":"Re: Conditional Formatting (modified date not in the last 3 weeks)","url":"https:\/\/community.smartsheet.com\/discussion\/comment\/392553#Comment_392553","dateInserted":"2023-08-25T12:40:57+00:00","insertUserID":161267,"insertUser":{"userID":161267,"name":"Jef Snyders","title":"Jef Snyders","url":"https:\/\/community.smartsheet.com\/profile\/Jef%20Snyders","photoUrl":"https:\/\/us.v-cdn.net\/6031209\/uploads\/userpics\/4HJAEW33KBD0\/nXEKEZE5EQEV4.jpg","dateLastActive":"2023-08-25T13:32:31+00:00","banned":0,"punished":0,"private":false,"label":"✭✭"}},"breadcrumbs":[{"name":"Home","url":"https:\/\/community.smartsheet.com\/"},{"name":"Get Help","url":"https:\/\/community.smartsheet.com\/categories\/get-help"},{"name":"Smartsheet Basics","url":"https:\/\/community.smartsheet.com\/categories\/smartsheet-basics%2B"}],"groupID":null,"statusID":3,"attributes":{"question":{"status":"accepted","dateAccepted":"2023-08-25T12:40:09+00:00","dateAnswered":"2023-08-25T12:37:47+00:00","acceptedAnswers":[{"commentID":392551,"body":"
There is no direct way to do this. Adding a checkbox helper column with something similar to this and then formatting based on the checkbox is the simplest way. You can even hide the extra column, as there is no reason for it to be visible.<\/p>
=IF([Modified Date]@row < TODAY(-21), 1, 0)<\/p>"}]}},"status":{"statusID":3,"name":"Accepted","state":"closed","recordType":"discussion","recordSubType":"question"},"bookmarked":false,"unread":false,"category":{"categoryID":321,"name":"Smartsheet Basics","url":"https:\/\/community.smartsheet.com\/categories\/smartsheet-basics%2B","allowedDiscussionTypes":[]},"reactions":[{"tagID":3,"urlcode":"Promote","name":"Promote","class":"Positive","hasReacted":false,"reactionValue":5,"count":0},{"tagID":5,"urlcode":"Insightful","name":"Insightful","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":11,"urlcode":"Up","name":"Vote Up","class":"Positive","hasReacted":false,"reactionValue":1,"count":0},{"tagID":13,"urlcode":"Awesome","name":"Awesome","class":"Positive","hasReacted":false,"reactionValue":1,"count":0}],"tags":[{"tagID":437,"urlcode":"conditional-formatting","name":"Conditional Formatting"}]}],"initialPaging":{"nextURL":"https:\/\/community.smartsheet.com\/api\/v2\/discussions?page=2&categoryID=321&includeChildCategories=1&type%5B0%5D=Question&excludeHiddenCategories=1&sort=-hot&limit=3&expand%5B0%5D=all&expand%5B1%5D=-body&expand%5B2%5D=insertUser&expand%5B3%5D=lastUser&status=accepted","prevURL":null,"currentPage":1,"total":4991,"limit":3},"title":"Trending in Smartsheet Basics","subtitle":null,"description":null,"noCheckboxes":true,"containerOptions":[],"discussionOptions":[]}">
