Page 1 of 3 123 LastLast
Results 1 to 10 of 22

Thread: Managing large amounts of electronic files (photos)

  1. #1
    Site Supporter LOKNLOD's Avatar
    Join Date
    Feb 2011
    Location
    Oklahoma

    Managing large amounts of electronic files (photos)

    Anybody have some pro-tips on managing large quantities of files - photos in particular, if that adds any options for specialized tools.

    My wife loves photography. Consequently we have 2.3 metric craptons of digital photography files. So much that when we backed up her phone the folder has so much in it that my computer doesn't even like opening the folder. And of course, there are some extremely special photos of life events sprinkled in this haystack, so wiping them and starting fresh isn't an option, and the thought of losing them permanently is bad too.

    Surely, there's a better way. I've yet to find any automated backup stuff that doesn't drive me fucking crazy with filling itself up and not properly overriding old info.

    We're all apple for phones and all windows for computer, if that helps/hurts any.
    --Josh
    “Formerly we suffered from crimes; now we suffer from laws.” - Tacitus.

  2. #2
    "Digital Asset Management" (yes DAM) is a yuge area of endeavor and although I find it interesting it can be complicated and time consuming.

    One of the big things in the DAM world is categorizing, organizing, tagging, culling, etc. to add / maintain the value / convenience of your photos. This is one of the time consuming parts. Specifically with regard to photos there are programs such as Lightroom that provide ways to do all of this and more, again I find it interesting but paying for it, installing it, learning how to use it, actually using it consistently, etc. are not without a cost in time and treasure.

    For backup I am using Network Attached Storage which is a (small) computer with a number of hard drives that provide for re-generating any one after a failure. This system can be used to back up PC and Macs which are what I mainly use for images and other work. My phone is backed up to iCloud which is not ideal but is very convenient especially if the phone itself is lost, stolen or damaged.

    And it goes on and on from there especially as the value (commercial or personal) of your digital stuff increases. Depending on who is talking "best practices" include a combination of regular, automated backups kept onsite and periodic backups that are stored offsite, sometimes augmented by cloud backups especially for travel. Al of which needs to be set up, tested and maintained.

    None of it is super hard but there are a lot of things to know and do, and I suspect most of us are not doing all of them properly.

  3. #3
    Chasing the Horizon RJ's Avatar
    Join Date
    Jan 2014
    Plus one on a NAS / RAID. I haven't had a need for one in a while, and I'm sure the consumer market is flooded with options.

    At work (now retired) we had to have a fair old amount of data for post-install of simulation systems that some pretty complex visual and terrain databases...like, a lot of data.

    My info is getting out of date, so I would defer to the folks closer to good affordable home solutions. Actually I'll be interested in what's suggested, as I may pick up a NAS to hang off my network at some future point as well.

  4. #4
    I took a lot of photos at my last job. Inventory photos for an offroad dealership. They made the choice to use internal PC storage and external hard drives. They were also saved online (not really cloud... more just web based) when uploaded. If properly organized, it's doable.

    The hard drive organizing wasn't to bad, but a clear cut scheme is important. I used Year - Make - Model - Color - Options as my layout. If you can find a layout that works for what your taking photos of you'll know where to go tofind what you want.

    The online saving was through the dealership's website host. It was clunky, but provided a backup that was accessible from more than one PC. I wasn't privy to billing, but I know the company paid through the nose.

  5. #5
    Smoke Bomb / Ninja Vanish Chance's Avatar
    Join Date
    Nov 2011
    What has already been mentioned are great ideas. The only thing I might suggest is incorporating the date the photo was taken in the file name in a "year-month-day" format (e.g., 2020-10-05), as I've lost metadata on photos after moving them around. I know what's in the photo just by looking at it, but I don't know when it was taken, so putting things in chronological order is major pain. I'm sure with the volume of photos you have, that'd be a massive investment of time. But after having helped clear closets after someone's passing, there is nothing more depressing than finding a photo and having no idea who is in it or when/where it was taken.

    You can look at NAS if you'd like, but some of the auto-backup solutions I've tried are finicky. Small sample size and so forth, but it never became a "set it and forget" scheme for me. I use Backblaze for long-term offsite backup. There's plenty of other great solutions out there, just make sure your wife and family have a way of getting into those accounts in case you're eaten by a meteorite or something.
    "Sapiens dicit: 'Ignoscere divinum est, sed noli pretium plenum pro pizza sero allata solvere.'" - Michelangelo

  6. #6
    Site Supporter LOKNLOD's Avatar
    Join Date
    Feb 2011
    Location
    Oklahoma
    When she’s taken the time to do it in the past, she did a great job and has a pretty good handling on how she wants them grouped and organized. Our old stuff looks great. But I just dumped her phone to back up and there are 27,000 files in the folder now. My computer is pooping itself just trying to open and index the folder. Ugh.
    --Josh
    “Formerly we suffered from crimes; now we suffer from laws.” - Tacitus.

  7. #7
    The R in F.A.R.T RevolverRob's Avatar
    Join Date
    May 2014
    Location
    Gotham Adjacent
    As folks said, data management is a full-time job in many respects.

    Add me to the NAS/RAID camp.

    Here in Casa de RevólverRoberto we use a DROBO 5N which currently has ~2TB of storage available and 2TB of storage used. Mind you, I back up more than photos, but a lot of what I do have are stacks of images (3D scans of things) and those stacks cannot be separated from one another, so it's important to maintain integrity. Redundant storage with RAID backups is the best way to make sure you do not lose the data in question.

    NOW the challenging part is managing it.

    In my opinion, there is no single program out there that manages photos particularly well. Unless you're handy at writing code to analyze the file's metadata, you'll probably have to sort photos by hand and/or in batches by date.

    EDIT: I see you're running windows boxes at home. Man...I'm almost tempted to tell you to spend the 800-1000 bucks buying a Mac Mini or Macbook Air. Mac definitely handles importing and sorting of iPhone photos in a more efficient way. See below:

    [Mac Specific Stuff]

    If you're running a Mac and the photos come from an iPhone, I would simply start by importing them all into iPhoto (I guess it's just called 'Photos' now). There you can rather easily bulk select photos (hold shift to select things in a row/adjacent, or command and click photos individually to select multiples), right click (or use the Image drop down) and you can add them to an existing or create a new album. It won't be the fastest way to do these things ever, but you can speed the process up by processing things by date (click years/months/whatever at the top of Photos). There you can at least select on photos from a given year/period of time to sort through.

    Once added to an album, Photos updates on the back-end by creating a folder for each Album. Those can easily be migrated over to your NAS and accessed remotely or easily through a few clicks. Photos doesn't change the image format, so you end up with sorted folders of jpegs. Name the albums with date/event and you've got yourself reasonably well sorted.

    [/Mac Specific Stuff]

    It will take a couple of days to sort through thousands of photos this way. Your wife took the pictures, so she's the one who should sort them.

    ___

    When I'm doing these types of things for pictures of specimens, 3D scans, PDFs related to projects, etc. It's the kind of 'busy work' I have a brand new undergraduate volunteer do. It's boring, stupid, and time consuming. It's worth the effort in the end to find things quickly and efficiently. You have the misfortune of starting with a big pile of data and no organization. But you have the fortune of not having inherited someone else's goofy naming/organizational system (trust me that's a fucking nightmare you don't want to deal with). Data management, data bases, and absolute metric tons of metadata is a surprisingly large part of my job.

    ___

  8. #8
    Site Supporter LOKNLOD's Avatar
    Join Date
    Feb 2011
    Location
    Oklahoma
    Quote Originally Posted by RevolverRob View Post
    . But you have the fortune of not having inherited someone else's goofy naming/organizational system (trust me that's a fucking nightmare you don't want to deal with).
    (Sorta) Sidebar...
    I get dragged into data governance stuff at work from time to time (I am an enginerd, after all) and I agree
    I’m much happier starting with minimal org structure than a terrible one.

    My first job had a drawing file naming convention that predated the company having computers, so things were organized hyper logically so you could find old scans of hand drawn prints lickety split by tracing the convention, at least for the old stuff.

    The new stuff, they had a database, and drawing files were a sequential number and letter based on paper size if you plotted. Without the database... you were well a truly fucked finding anything.
    --Josh
    “Formerly we suffered from crimes; now we suffer from laws.” - Tacitus.

  9. #9
    Quote Originally Posted by LOKNLOD View Post
    Anybody have some pro-tips on managing large quantities of files - photos in particular, if that adds any options for specialized tools.

    My wife loves photography. Consequently we have 2.3 metric craptons of digital photography files. So much that when we backed up her phone the folder has so much in it that my computer doesn't even like opening the folder. And of course, there are some extremely special photos of life events sprinkled in this haystack, so wiping them and starting fresh isn't an option, and the thought of losing them permanently is bad too.

    Surely, there's a better way. I've yet to find any automated backup stuff that doesn't drive me fucking crazy with filling itself up and not properly overriding old info.

    We're all apple for phones and all windows for computer, if that helps/hurts any.
    Good tips thus far.

    DAM systems are useful for automating workflows but may not be worth the effort unless you use the images for something other than simple love of photography.

    The first thing I’d do is brutally edit your inventory. When I was a working photographer, professionals expected to get one usable image per 36-exposure roll and hoped to get one show-stopper per 15-20 rolls. Yes, I am that old but the concept remains valid. Most of these folks never looked back through their old images—there was almost never anything usable. If you have eight pictures of the same sunset/flower/landscape, chances are that seven of them can go. Actually, all eight can probably go unless they document some truly magical event.

    Zap everything that isn’t magical and your problem gets a lot easier. As you go through this process, categories will suggest themselves and you can go from there.


    Okie John
    “The reliability of the 30-06 on most of the world’s non-dangerous game is so well established as to be beyond intelligent dispute.” Finn Aagaard
    "Don't fuck with it" seems to prevent the vast majority of reported issues." BehindBlueI's

  10. #10
    Member
    Join Date
    Mar 2011
    Location
    FL
    I like using Google photos (granted, my phone is my camera). I find that gPhotos does a good job of sorting images by dates and storing them for me. Also, the photos go on the cloud automatically. I believe this would work even with Apple devices. The only caveat is that it doesn't store the photos in the native ultra high resolution but cuts it down to something else.

User Tag List

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •