URGENT | Need to delete items from project store


#1

I need your application to do it’s job and clean up after itself. I have a project that should only have about 75 assets according to the current.json yet the store folder has 2,887 files. What takes multiple hours to publish and says is only 29.4mb is actually a 371mb .zpp when the publish is downloaded. The store folder is now 822mb of unused data that is still being published.

I need a way to purge commit and unused assets from the store folder.


Clean your project before publishing!
#2

Yeah… i’m having the same issue here.
Actual file is 25mb or so, but the export is 200mb!

It is so anoying have to clean up de json file all by hand.


#3

Yeah I am going to lose half a day trying to fix their horrible design of hiding deleted assets from the Editor while leaving them a part of the store folder.

My current plan is open the current.json file, do a regex replace of ‘,’ with ‘\r\n’ (don’t include the '), then search the %userprofile%.zapparstudio~PROJECT~\store folder for the GUID files listed after “:”, move those to another folder, and delete what is left. Then search the structure.json GUID files listed in the current.json for etag, any GUID listed in those files needs to be moved back. Now go back through the current.json file and delete any line containing GUID files not moved back. Finally regex replace in the current.json ‘\r\n’ with ‘,’ and save.

I am also doing a test publish which will take probably 2-3 hours due to this issue. I will be posting back my findings. This seems to be a very bad design on the ZapWorks side and could be serious trouble for any user on metered internet as you are wasting upload and download for no reason.

-Eric


#4

Statistics from first pass purge of store folder based on current.json entries.

Original store folder File Count: 2,872
Original store folder File Size: 801 MB
Purged store folder File Count: 77
Purge store folder File Size: 385 MB


#5

Purge based on structure.json files.

Purged store folder File Count: 43
Purge store folder File Size: 67.5 MB


#6

Original project Time: 24:00:00+ *killed after 24 hours, never finished
Original project Reported Approx Size: unknown *killed after 24 hours, never finished

Purged project Time: 00:01:30
Purged project Reported Approx Size: 29.9 MB


#7

It took over an hour, but will save a ton of time in future publishing. Here is the final purge process I used.

WARNING: THIS WILL INVALIDATE AND REMOVE YOUR COMMIT HISTORY
WARNING: DO AT YOUR OWN RISK ON A COPY OF THE ORIGINAL PROJECT
WARNING: I AM SHARING PERSONAL RESULTS I AM NOT RESPONSIBLE IF YOU BREAK YOUR PROJECT WHILE TRYING THIS

  • Open %userprofile%\.zapparstudio\~PROJECT~\current.json in Notepd++
  • Do a regex replace (without the ’ ‘) Find:’,’ Replace:’\r\n’
  • Create a new folder in the project named store.backup
  • Find GUID names after every ‘:’ in the current.json and move from store to store.backup
  • Delete all files left in the store folder
  • Move all structure.json and editor.json UUID files from store.backup to store
  • Open all the structure.json UUID files in Notepad++ search for all etag and move all UUID files from store.backup to store
  • Now we need to remove all items not moved from the store.backup from the current.json file
  • Go one by one copy the name from explorer, Find in Notepad++, delete all found lines
  • Once all unused files have been removed from the current.json we need to return it to the original 1 line form
  • Regex Find:’\r\n’ Replace:’,’
  • Check to ensure there are no extra , and save the file.
  • Open the project and publish
  • Delete all dated .json files (YYYY-MM-DD-HH-MM-SS.json) and the snapshots.json

#8

I stopped the clock on the Original Publish as it has now hit 9 hours. I will let it run over night and see if it finishes.


#9

That doesn’t seem right. Has it been reported as an issue?


#10

Yes, with a first reply of:

As for your question, I’d recommend that you take a look at our ‘Optimizing your experience’ article here, going over how to remove assets from your library and other methods of reducing bloat.

However, this is worthless to this issue as it will only add more data that is being kept around and unused. For example, in the article it shows you how to use the image properties to make a smaller file, but unknown to the user both files are kept. So if you have a 10MB image that you use Image Properties to optimize down to 2MB, a normal person would assume that the 2MB shown in the Final Download Size tool is correct. However, in reality you increased your file size usage by 2MB to 12MB instead of reducing it by 8MB to 2MB. Then there is the issue that the file size reported after Publishing comes from the Final Download Size tool, not the actual size of what is uploaded. Even worse is if you instead replace the 10MB image with a 5MB image, and then replace that with a 2MB image your upload size is now 17MB not 2MB.

I just tested and verified what I noted above. Anything ever added into the Media Library will always be uploaded, but does not show in the Approx. Size or Final Download Size. Even if you replace or delete it from the Media Library it will still be uploaded.

24 hours later after I told them I was going to have to manually fix this issue they responded with:

Sorry for the inconvenience regarding your project’s size.

We’ve been able to compress (crush) your project ZPP down to 63MB using a method which we hope to make available via ZapWorks in the future to help clean projects, which should ensure faster publishing times for you.

A quick look at the zpp they sent and they did what I outlined here. I seriously hope this comes as a standalone tool ASAP and not as a 7.x feature.

-Eric


#11

Well, that sure explains why it takes longer than expected to upload my project which is less than a megabyte download size. :neutral_face:


#12

Scary part is that is not the real download size for the app. Go to the Project Overview on the website, in the Publish Log area click on SEE ALL LOG ENTRIES, and press DOWNLOAD on the item at the top of the list. That is the real download size for the project.

-Eric


Clean your project before publishing!
#13

What the… Holy!@#$! It’s actually 8 times bigger than ZapWorks studio says it is! This has got to be a bug!

So I’ve been following all the rules and guidelines to no avail???!!! I’m absolutely dumbfounded as to why this hasn’t been fixed. So I guess I need to learn how to squander my time manually optimizing the download size? Where is this process documented? This is complete nonsense.

For anyone stumbling upon this thread, my concerns were completely unjustified and based on erroneous information. Please see this post by @simon, which sets the record straight.


#14

Where are these files and folders located? Where is this manual “clean-up” process documented?

EDIT

Still being somewhat new to ZapWorks, I hope I’m missing something here; but all this time, I thought users were downloading less than a megabyte over the cell network. If I understand this thread, though, it seems they’ve actually been downloading about 8 times that. (That’s the scale of the discrepancy between what ZapWorks Studio reports and the size of the ZPP file in my account’s upload log.)

Do I understand correctly?

For anyone stumbling upon this thread, my concerns were completely unjustified and based on erroneous information. Please see this post by @simon, which sets the record straight.


#15

Correct, the .zpp file you downloaded is what is downloaded and unpacked during the scan “unlocking”.

The path is the first line in my instructions in this thread. For Windows the location of your projects is %userprofile%\.zapparstudio\


#16

Thanks. Is there any reason the clean-up process you describe couldn’t be done on an exported (or downloaded) copy of the project so as not to muck with the commit history etc?


#17

And for Mac, it’s at ~/.zapparstudio. (The tilde means the user’s home directory - e.g. Users/jane - and the period at the beginning of the file name means it’s a hidden directory, so you won’t see it in a “normal” Finder listing.)


#18

Sure just an added step to go that route. The .zpp appears to be a zip copy of the project, so duplicate the folder and save the time downloading and unpacking.

Part of the reason they keep all of the files is they are needed for the commit history. If you are deleting items from the store the commit history will fail to load if a file is missing.


#19

Thanks. I just recently posted a question asking how to delete commits. I don’t need the entire history. It grows too rapidly and sucks up too much storage on my backup drive.

Do you know if there’s a way to selectively delete commits?


#20

The commits are just the dated .json files in the project root. The commit files are small and it is the referenced asset files in the “store” folder that hold all the data. You would have to manually review .json files to figure out what you need to keep.