Category: Software Development

Efficiently handling large Eloquent results in Laravel

   

We sometimes need to handle large results of models when working with Eloquent. If we use the regular method of retrieving the full result set and working with the models, then it’s going to overflow our memory – essentially failing the entire process.

In order to bring efficiency to that process, we can lazy load the models in chunks and process them. Laravel provides two methods for that: lazy() and lazyById().

Consider a scenario where in a console command, we need to update a course’s average rating (example from Laravel Courses). Here’s how we can do that using the lazy() method, where courses will be fetched in chunks, and we can work with each of the retrieved models individually:

use App\Models\Course;
 
foreach (Course::lazy() as $course) {
    $course->updateAverageRating();
}

Consider another use case where we want to perform bulk updates to a model but want to do that in chunks. Let’s see how that can be achieved: 

use App\Models\Invoice;
 
Invoice::where('status', 'pending')
    ->lazyById(100)
    ->each->update(['status' => 'abandoned']);

The official documentation also has some good examples you can look into as well.

Choosing Boring Technology

A wonderful presentation by Dan McKinley of Stripe on how technology companies make mistakes in choosing the next piece of shiny technology instead of relying on their existing and tested stack.

He outlines how this increases the overall cost and how this cripples the development team in many ways. I have to admit that I used to do this myself for many years and would go for the next framework, the next data store, etc.

I learned the lesson the hard way, and thankfully since then, I have chosen the fewer, the most known technology that serves the need.

Concurrent HTTP requests in PHP using pecl_http

 

The pecl_http extension has a little gem that can be handy at times – HttpRequestPool. Using this, you can send concurrent HTTP requests and can gain efficiency in fetching non-related data at once. For example, from an external source if your application needs to retrieve an user’s profile, their order history and current balance, you can send parallel requests to the API and get everything together.

Here is a simple example code to illustrate that:

 1<?php
 2 
 3$endpoint = "http://api.someservice.com";
 4$userId = 101;
 5 
 6$urls = array(
 7    $endpoint . '/profile/' . $userId,
 8    $endpoint . '/orderHistory/' . $userId,
 9    $endpoint . '/currentBalance/' . $userId
10);
11 
12$pool = new HttpRequestPool;
13 
14foreach ($urls as $url) {
15    $req = new HttpRequest($url, HTTP_METH_GET);
16    $pool->attach($req);
17}
18 
19// send all the requests. control is back to the script once 
20// all the requests are complete or timed out
21$pool->send();
22 
23foreach ($pool as $request) {
24    echo $request->getUrl(), PHP_EOL;
25    echo $request->getResponseBody(), PHP_EOL . PHP_EOL;
26}

Amazon S3 integration with Symfony2 and Gaufrette

Its amazing how Symfony2 has created an ecosystem where you can just add a bundle to your application and in minutes, you are able to make use of functionality/integration that would take you days to add if you had to code it up yourself. A while ago, I needed to add S3 CDN support to our Loosemonkies application so that the avatars uploaded by job seekers and company logos uploaded by employers would be stored in a globally accessible CDN. I started looking for a ready-to-use bundle that I can just add to the project, beef up a few config and everything else would just work. However, this time the task seemed a bit tough.

After much searching and evaluating, I stumbled upon this wonderful post. The author has shown here how to use Gaufrette and the Amazon AWS bundle together to achieve what I was looking for. It took me a while to follow the steps and finally I integrated it. Then I had a new requirement where I would need to upload assets that have already been sitting in the local, so I would add an additional method in the PhotoUploader class to handle it – uploadFromUrl. It would guess the mime type of the file by extension, as we do not have the mime type handed to us by PHP. It worked beautifully and we were all set.

Sharing the code here in case others find it useful.

 1<?php
 2 
 3namespace LM\Bundle\CoreBundle\Controller;
 4 
 5use Symfony\Component\HttpFoundation\Request;
 6use Symfony\Component\HttpFoundation\Response;
 7use Symfony\Bundle\FrameworkBundle\Controller\Controller;
 8 
 9class AppController extends Controller
10{
11    /**
12     * Upload Image to S3
13     *
14     * @param string $name Image field name
15     * @param int $maxWidth Maximum thumb width
16     * @param int $maxHeight Maximum thumb height
17     *
18     * @return string
19     */
20    protected function uploadImage($name, $maxWidth = 100, $maxHeight = 100)
21    {
22        $image = $this->getRequest()->files->get($name);
23 
24        $uploader = $this->get('core_storage.photo_uploader');
25        $uploadedUrl = $uploader->upload($image);
26 
27        return $this->container->getParameter('amazon_s3_base_url') . $uploadedUrl;
28    }
29}
1{

2    "require": {

3        "knplabs/gaufrette": "dev-master",

4        "knplabs/knp-gaufrette-bundle": "dev-master",

5        "amazonwebservices/aws-sdk-for-php": "dev-master"

6    }

7}
 1imports:

 2    - { resource: parameters.yml }

 3    - { resource: security.yml }

 4    - { resource: @LMCoreBundle/Resources/config/services.yml }

 5

 6knp_gaufrette:

 7    adapters:

 8        photo_storage:

 9            amazon_s3:

10                amazon_s3_id: loosemonkies_core.amazon_s3

11                bucket_name:  %amazon_s3_bucket_name%

12                create:       false

13                options:

14                    create: true

15    filesystems:

16        photo_storage:

17            adapter:    photo_storage

18            alias:      photo_storage_filesystem

19

20loosemonkies_core:

21  amazon_s3:

22    aws_key: %amazon_aws_key%

23    aws_secret_key: %amazon_aws_secret_key%

24    base_url:       %amazon_s3_base_url%
 1<?php
 2 
 3namespace LM\Bundle\CoreBundle\DependencyInjection;
 4 
 5use Symfony\Component\Config\Definition\Builder\TreeBuilder;
 6use Symfony\Component\Config\Definition\ConfigurationInterface;
 7 
 8/**
 9 * This is the class that validates and merges configuration from your app/config files
10 *
11 * To learn more see {@link http://symfony.com/doc/current/cookbook/bundles/extension.html#cookbook-bundles-extension-config-class}
12 */
13class Configuration implements ConfigurationInterface
14{
15    /**
16     * {@inheritDoc}
17     */
18    public function getConfigTreeBuilder()
19    {
20        $treeBuilder = new TreeBuilder();
21        $rootNode = $treeBuilder->root('loosemonkies_core');
22 
23        // Here you should define the parameters that are allowed to
24        // configure your bundle. See the documentation linked above for
25        // more information on that topic.
26 
27        $rootNode
28            ->children()
29                ->arrayNode('amazon_s3')
30                    ->children()
31                        ->scalarNode('aws_key')->end()
32                        ->scalarNode('aws_secret_key')->end()
33                        ->scalarNode('base_url')->end()
34                    ->end()
35                ->end()
36            ->end();
37 
38        return $treeBuilder;
39    }
40}
1# This file is auto-generated during the composer install

2parameters:

3    locale: en

4    secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

5    amazon_aws_key: XXXXXXXXXXXXXXXXXXXX

6    amazon_aws_secret_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX+

7    amazon_s3_bucket_name: dev-assets

8    amazon_s3_base_url: 'http://s3.amazonaws.com/dev-assets/'
  1<?php
  2 
  3namespace LM\Bundle\CoreBundle\Service;
  4 
  5use Symfony\Component\HttpFoundation\File\UploadedFile;
  6use Gaufrette\Filesystem;
  7 
  8class PhotoUploader
  9{
 10    private static $allowedMimeTypes = array('image/jpeg', 'image/png', 'image/gif');
 11    private $filesystem;
 12 
 13    public function __construct(Filesystem $filesystem)
 14    {
 15        $this->filesystem = $filesystem;
 16    }
 17 
 18    public function upload(UploadedFile $file)
 19    {
 20        // Check if the file's mime type is in the list of allowed mime types.
 21        if (!in_array($file->getClientMimeType(), self::$allowedMimeTypes)) {
 22            throw new \InvalidArgumentException(sprintf('Files of type %s are not allowed.', $file->getClientMimeType()));
 23        }
 24 
 25        // Generate a unique filename based on the date and add file extension of the uploaded file
 26        $filename = sprintf('%s/%s/%s/%s.%s', date('Y'), date('m'), date('d'), uniqid(), $file->getClientOriginalExtension());
 27 
 28        $adapter = $this->filesystem->getAdapter();
 29        $adapter->setMetadata($filename, array('contentType' => $file->getClientMimeType()));
 30        $adapter->write($filename, file_get_contents($file->getPathname()));
 31 
 32        return $filename;
 33    }
 34 
 35    public function uploadFromUrl($url)
 36    {
 37        // Get file extension
 38        $extension = pathinfo($url, PATHINFO_EXTENSION);
 39 
 40        // Generate a unique filename based on the date and add file extension of the uploaded file
 41        $filename = sprintf('%s/%s/%s/%s.%s', date('Y'), date('m'), date('d'), uniqid(), $extension);
 42 
 43        // Guess mime type
 44        $mimeType = $this->guessMimeType($extension);
 45 
 46        $adapter = $this->filesystem->getAdapter();
 47        $adapter->setMetadata($filename, array('contentType' => $mimeType));
 48        $adapter->write($filename, file_get_contents($url));
 49 
 50        return $filename;
 51    }
 52 
 53    private function guessMimeType($extension)
 54    {
 55        $mimeTypes = array(
 56 
 57            'txt'  => 'text/plain',
 58            'htm'  => 'text/html',
 59            'html' => 'text/html',
 60            'php'  => 'text/html',
 61            'css'  => 'text/css',
 62            'js'   => 'application/javascript',
 63            'json' => 'application/json',
 64            'xml'  => 'application/xml',
 65            'swf'  => 'application/x-shockwave-flash',
 66            'flv'  => 'video/x-flv',
 67 
 68            // images
 69            'png'  => 'image/png',
 70            'jpe'  => 'image/jpeg',
 71            'jpeg' => 'image/jpeg',
 72            'jpg'  => 'image/jpeg',
 73            'gif'  => 'image/gif',
 74            'bmp'  => 'image/bmp',
 75            'ico'  => 'image/vnd.microsoft.icon',
 76            'tiff' => 'image/tiff',
 77            'tif'  => 'image/tiff',
 78            'svg'  => 'image/svg+xml',
 79            'svgz' => 'image/svg+xml',
 80 
 81            // archives
 82            'zip'  => 'application/zip',
 83            'rar'  => 'application/x-rar-compressed',
 84            'exe'  => 'application/x-msdownload',
 85            'msi'  => 'application/x-msdownload',
 86            'cab'  => 'application/vnd.ms-cab-compressed',
 87 
 88            // audio/video
 89            'mp3'  => 'audio/mpeg',
 90            'qt'   => 'video/quicktime',
 91            'mov'  => 'video/quicktime',
 92 
 93            // adobe
 94            'pdf'  => 'application/pdf',
 95            'psd'  => 'image/vnd.adobe.photoshop',
 96            'ai'   => 'application/postscript',
 97            'eps'  => 'application/postscript',
 98            'ps'   => 'application/postscript',
 99 
100            // ms office
101            'doc'  => 'application/msword',
102            'rtf'  => 'application/rtf',
103            'xls'  => 'application/vnd.ms-excel',
104            'ppt'  => 'application/vnd.ms-powerpoint',
105            'docx' => 'application/msword',
106            'xlsx' => 'application/vnd.ms-excel',
107            'pptx' => 'application/vnd.ms-powerpoint',
108 
109            // open office
110            'odt'  => 'application/vnd.oasis.opendocument.text',
111            'ods'  => 'application/vnd.oasis.opendocument.spreadsheet',
112        );
113 
114        if (array_key_exists($extension, $mimeTypes)){
115            return $mimeTypes[$extension];
116        } else {
117            return 'application/octet-stream';
118        }
119 
120    }
121 
122}
 1parameters:

 2

 3  core.amazon_s3.class: AmazonS3

 4  core_storage.photo_uploader.class: LM\Bundle\CoreBundle\Service\PhotoUploader

 5

 6services:

 7

 8  loosemonkies_core.amazon_s3:

 9    class: %core.amazon_s3.class%

10    arguments:

11         - { key: %amazon_aws_key%, secret: %amazon_aws_secret_key% }

12

13  core_storage.photo_uploader:

14    class: %core_storage.photo_uploader.class%

15    arguments: [@photo_storage_filesystem]