5

I'm creating a video sharing website in django.

Currently, if a user registers, and upload a video, it will be uploaded to media/vid/uploaded-vid. Then i converted using ffmpeg to flv. What i would like to do is this:

Someone with the username of alex1 registers, i would like to create a directory for him when he confirms his email, called /media/vid/members-vid/alex1

If he uploaded a video, it will be converted to flv in media/vid/uploaded-vid then copied to /media/vid/members-vid/alex1. And the video in media/vid/uploaded-vid should be deleted.

And I would like to secure /media/vid/. How do you secure django directories? Or is just an apache chmod?

i was wondering if i can use celery/rabbitqm to copy files from one folder to another, or to create new folders..

user
  • 973
  • 4
  • 14
  • 29
  • @mikerobi from hackers, I want to know what is the most secure way to store the videos on my server – user Apr 02 '12 at 18:50
  • I meant, are you trying to protect someone from accessing a private video over the web, or if someone gains access to your server? – mikerobi Apr 02 '12 at 19:06
  • @mikerobi both, i don't want to let someone access my own server, and accessing private videos, i wasn't thinking about private videos when i asked this question, but now that you mentioned it, i want to prevent these 2 scenarios from happening – user Apr 02 '12 at 19:26
  • 2
    Sorry but I think your question is overly broad & vague, and probably won't be answered. For the record, creating a directory is really easy if you already have the user registration page: http://stackoverflow.com/questions/273192/python-best-way-to-create-directory-if-it-doesnt-exist-for-file-write And BTW, I think concurrency issues can raise if you encode all the videos on the same directory. – Mariano Desanze Apr 04 '12 at 23:02

3 Answers3

5

To avoid someone uploading a shell or some other malicious code you should not give access to uploaded file to the outside clients. So user uploads a file, than celery gets that file and processes it, putting the result to another, webserver accessible path. If processing fails (i.e. file is not a valid video), then noone can access it. But with correct webserver setup (i.e. deny running any scripts from locations availble for upload) it should be no big problem anyway.

To avoid users see other's private files, you can also put the files outside of site's mediafiles, and use django's views for checking access rights + special webserver directive to actually server the file without proxying it through Django. These directives are different for diffeent servers:

ilvar
  • 5,718
  • 1
  • 20
  • 17
  • 1
    I recommend using the [django-sendfile](https://github.com/johnsensible/django-sendfile) module to use the file proxying features. – Danilo Bargen Apr 09 '12 at 18:11
3

Your use of a holding directory seems to be a good one. However having directories named after users is risky (what if someone provides a name which is an executable command name?) so using the user id might be better. Indeed if you are going to have lots of users you might want to consider splitting up user directories under some general directories to make finding the directories easier for admin purposes, and because some file systems have rather low limits on the number of entries under a directory -- e.g. ext3 allows roughly 32K entries.

For instance, a crude user directory scheme with outer directories named 1 to 9 for subdirectories starting with those numbers will give you a bit more flexibility here.

So, assuming you have your uploaded files in /tmp/upload/1015/, and you want to move them to /var/userdata/01/1015/ and process files in /var/userdata/01/1015/, the following might be a sensible approach:

  • /tmp/upload/1015/ will need to be apache user or group writeable
  • record the need to process files in /tmp/upload/1015/ in a database, AMQP scheduling service such as RabbitMQ or an RPC/web services call
  • a scheduling service client running under its own userid picks up the job from db/AMQP/web call and does the following
    • run a virus check
    • runs a file sanity check to see if the source file is what it expects
    • copies the file to the user folder
    • if successful, removes the original file (or schedules a job to do that if it does not have permission to do so, or relies on a cronjob to delete files in /tmp/upload that are more than 24 hours old)
    • attempts to convert the files, updating relevant success/failure flags in the database as it goes along
    • deletes temporary files/unsuccessful source/output files on completion

The list obviously would go on further for a while, depending on your needs. In the end you will have a user directory with files readable by apache but not writeable by the apache user. If your service grows you can move the video processing components of this service to another server.

By the way there is a very good description of how to do this sort of thing (very simply) by Cal Henderson of Flickr in his O'Reilly book "Building Scalable Web Sites". It was written in 2006 but his approach is refreshingly direct and straight-forward.

rorycl
  • 1,344
  • 11
  • 19
  • +1. This is a good answer. One more piece of advice though: the file conversion step is going the be the part of the processing that executes the most code, and this code will also be complex third-party code (codecs and so on). Which precise code gets executed will be under the end-user's control (since they choose the file type) and so this code should be among the last privileged parts of the code. Certainly, don't let it run with privileges to read the database, let alone write to it. – James Youngman Apr 08 '12 at 07:33
1

May be you should store your videos in mongoDB GridFS or something similar. So you will not have any trouble with directories, storage, executable files ...

Tewfik
  • 176
  • 2
  • 15