Creating the new mnjul.net homepage

於是我又~炸掉我的英文個人首頁,然後重做一個了。其實上一版也是一年前才做好( 算是活得最短的一個版本 ),但當時有點是為了找工作,所以急就章搞了一個也不知道到底要放什麼東西、要做什麼功能、要用什麼方式呈現、要怎樣設計的怪東西。前陣子總算有時間想 & 實際操作,就開始動工。

 

英文個人首頁,到底要放什麼東西?

自從六年前的橫向捲動的版本開始,一直以來我都把我的英文個人首頁當做 portal,放入( 幾乎是所有的 )我做的東西的連結,了不起再分門別類一下,最後加個 about me 的小 pop-up 就收工。也就是說,乍聽之下,根本就是作品集--但又是大雜匯的作品集( 姑且稱之為 meta-作品集 ):網站作品、程式碼作品、攝影作品、音樂作品、寫作作品,但又有其他亂七八糟的連結( 連到我的 Twitter 之類 );這樣胡亂攪和起來,要如何簡單扼要地呈現就成為一大難題。如果想再對每個連結都添上文字介紹,就亂糟糟了。下圖是這次被我炸掉的去年的版本:雖然有各種縮圖,但還是眼花撩亂地,沒什麼結構感、層次感。

[ 去年的版本 ]

(繼續閱讀…)

Read More

伺服器( 更多 )HTTPS 支援

最近讓這台 server 有更多 domain 支援 HTTPS 了!

其實之前 Let’s Encrypt 在 beta 的時候,我就有參加,並且也有 deploy 到我的 server 上。不過我的 deployment 一直都沒有很廣;因為我一些 domain 都額外有由 CloudFront 提供的 CDN 支援,要把 HTTPS certificate 推上去諸多麻煩。前陣子雖然 Amazon 發表了 Certificate Manager,可以快速且自動申請 certificate,但那時候 domain validation 的信一直寄不到我的 protected whois 的 proxy email address,所以我也沒辦法用。直到一兩週前我發現突然信又寄得到了,所以我就很開心地開始大幅 deploy HTTPS 啦啦啦啦!

( 我這台 server 放了一~堆 domain,所以 HTTPS 都透過 SNI 處理 )

大部分的 domain 其實都還滿好操作的,就執行一下 Let’s Encrypt command line( 我用 --standalone ),然後把 apache conf 改一下;再去 ACM 申請一下 certificate,放到 CloudFront,就可以 deploy 了。不過我以前寫的網頁,很多外連結或資源都沒有用 protocol relative URL,瀏覽器就會吐 error 或 warning( 當然身為程式設計師,後者是不用管的 ),所以還得把一些 http:// 的東西換成 //

除了之前的 mnjul.net 跟 purincess.tw ,這次一口氣做了這些 domain:

  • (cdn.)allodynia.mnjul.net:一些 protocol relative 的東西要戳。
  • (cdn.)home.mnjul.net:意外地完全不用改任何 Flash 的 code,所以很好弄。
  • (cdn.)albums.purincess.tw:這個也滿簡單的。另外,piwigo 還有一個 CDN 外掛,所以就算裡面的照片要丟給 CDN serve,也不會太麻煩。這次還順便更新了很久沒更新的 piwigo。
  • hopperlive.mnjul.net:雖然沒在用,不過因為 charlie_su1986 大神幫我做的網站,實在捨不得拿下來,就…。
  • wretchalbum.purincess.tw:雖然沒在用了,但因為不用改任何 code( 包括 Flash 也不用改 ),所以很快可以搞定,就順便。
  • (cdn.)blogs.purincess.tw:這個就有點麻煩了。首先,apache conf 有一些額外要複製的東西。再者 wordpress 對於 base url 好像不太有 protocol relative 的概念。然後有些 theme/plug-in 沒有 protocol relative,得自己去戳檔案。然後還要去把資料庫文章裡面的一些網址改成 protocol relative:UPDATE wp_posts SET post_content = REPLACE(post_content, 'http://cdn.blogs.purincess.tw', '//cdn.blogs.purincess.tw'); 之類。

有一些不太能在這裡公開的 domain 就不特別說了。不過我現在只是初步開始支援 HTTPS,並沒有真的要做成 HTTPS first。大概還要之後很有空才能繼續弄吧。

Read More

Singleton Boto Client Wrappers In a Multi-Threaded Environment

Well, the title already sounds thrilling, I know. But that’s the deal today.

Here at Plurk, we use AWS DynamoDB to store some key-value things. This includes, for example, mapping from a single plurk’s ID into its database shard ID, and maintaining global (cross-shard) atomic counters. And we use boto to communicate with DynamoDB. Since boto3 wasn’t available (or rather, wasn’t declared as stable) when we transitioned to AWS, we use boto2. We probably would’ve needed less threading library constructs if we could utilize boto3 sessions!

Indeed, in our codebase, boto library calls are wrapped in our customized modules to expose semantically meaningful data/field accessors. Conventionally, most of our modules are singleton utilities (I’m personally not sure of exactly why, though). For example, if we want to dig out a user’s ID from their username, we’ll do a user_id = Users().get_id_from_name(username), with Users class extending (or rather, mixing-in) the Singleton base class.

 
Preface: (Thread-Safe) Singletons

For sake of completeness, here’s the implementation of that Singleton class:

(For brevity, imports are omitted, or things are used as were imported into the current namespace with from ... import ...)

class Singleton(object):
  __it_lock__ = threading.Lock()
  __it__ = None

  def __new__(cls, *args, **kwargs):
    if not cls.__it__:
      with cls.__it_lock__:
        if not cls.__it__:
          it = object.__new__(cls)
          it.init(*args, **kwargs)
          cls.__it__ = it
    return cls.__it__

  # Don't use __init__() because an instance returned
  # by __new__() will have its __init__() called
  # automatically every time __new__() is called. 
  @overridable
  def init(self, *args, **kwargs):
    pass

Alright, hope this metaclass-flavor singleton implementation is not too ugly, because this is actually more than a typical one — this singleton implementation is thread safe, guarded by double-checked locking. Why do we need thread safety? Because we use CherryPy as our HTTP server, where each HTTP request is served by a different thread, to attain the best possible throughput. So, when our server just launches, it immediately gets lots of requests, and there’s gonna be some occasion where singletons for the same class are almost simultaneously being requested (attribute that “almost” to Python’s GIL) — and we want to make sure only one has been instantiated in the end.

 
Wrapping Boto (and Failing)

Back to our main topic. To wrap our boto calls, taking the plurk mapping and atomic counter examples above, this was the first version of the modules I wrote…

class _DDBBase(Singleton):
  @override
  def init(self, *args, **kwargs):
    self._conn = boto.dynamodb2.connect_to_region(
      config.DYNAMODB_REGION,
      awesomes_access_key_id=config.DYNAMODB_ACCESS_ID,
      aws_secret_access_key=config.DYNAMODB_ACCESS_KEY
    )

class DDBPlurksMap(_DDBBase):
  @override
  def init(self, *args, **kwargs):    
    super(DDBPlurksMap, self).init(*args, **kwargs)
    self._ddb_plurks_map = DDBTable(config.DYNAMODB_TABLE_PLURKS_MAP, connection=self._conn) # from boto.dynamodb2.table import Table as DDBTable

  def get_shard_id(self, plurk_id, strong_consistent=False):
    try:
      item = self._ddb_plurks_map.get_item(consistent=strong_consistent, id=plurk_id)
    except ItemNotFound: # from boto.dynamodb2.exceptions import ItemNotFound
      return None
    else:
      return int(item['shard_id'])

class DDBAtomicCounters(_DDBBase):
  def _inc_and_get(self, key_name):
    res = self._conn.update_item(config.DYNAMODB_TABLE_ATOMIC_COUNTERS, {'name': {'S': key_name}}, {'number': {'Action': 'ADD', 'Value': {'N': '1'}}}, return_values='UPDATED_NEW')
    return int(res['Attributes']['number']['N'])

  def next_magic_id(self):
    return self._inc_and_get('magic_counter')

Looks properly abstracted? Yeah, it turned out this code worked… most of the time, but not always: These pieces of codes were not thread-safe, because boto2 was not thread-safe due to its use of httplib. So, as all my threads shared only one DynamoDBConnection instance stored at _DDBBase._conn, I was perfectly on my way to all the (not so) nice racing conditions down the road. (The dynamodb2.table instance using that connection was also shared, and was probably in peril, too).

 
Thread-Local Storage to the Rescue (Not Really, Yet)

To solve this with boto2, I’d need one connection for each thread. To do that, I needed a thread-local storage. Uhm… I tried writing something like this.

class _DDBBase(Singleton):
  @override
  def init(self, *args, **kwargs):
    self._tls = threading.local() # <- here it is
    self._tls.conn = boto.dynamodb2.connect_to_region(
      config.DYNAMODB_REGION,
      awesomes_access_key_id=config.DYNAMODB_ACCESS_ID,
      aws_secret_access_key=config.DYNAMODB_ACCESS_KEY
    )

class DDBPlurksMap(_DDBBase):
  @override
  def init(self, *args, **kwargs):    
    super(DDBPlurksMap, self).init(*args, **kwargs)
    self._tls.ddb_plurks_map = DDBTable(config.DYNAMODB_TABLE_PLURKS_MAP, connection=self._tls.conn)

  def get_shard_id(self, plurk_id, strong_consistent=False):
    try:
      item = self._tls.ddb_plurks_map.get_item(consistent=strong_consistent, id=plurk_id)
    except ItemNotFound:
      return None
    else:
      return int(item['shard_id'])

For sure, this didn’t work: Since Singleton.init() was called only once for a class across the Python process’s entire life time, only one thread — the one that successfully acquired the singleton lock and executed that init() — had the DynamoDBConnection instance (and the dynamodb2.table instance in DDBPlurksMap‘s case) in its thread local storage. All other threads, while having their own thread-local storage as self._tls, didn’t get those instances constructed and stored, and would hit an AttributeError once they tried to access the instances.

 
Ugly Solution

Indeed, I could write something like the following…

class _DDBBase(Singleton):
  @override
  def init(self, *args, **kwargs):
    self._tls = threading.local()
    self.reinit()

  @overridable
  def reinit(self):
    self._tls.conn = boto.dynamodb2.connect_to_region(
      config.DYNAMODB_REGION,
      awesomes_access_key_id=config.DYNAMODB_ACCESS_ID,
      aws_secret_access_key=config.DYNAMODB_ACCESS_KEY
    )

…and have subclasses check for the need to reinit() whenever a method was called…

class DDBPlurksMap(_DDBBase):
  @override
  def init(self, *args, **kwargs):    
    super(DDBPlurksMap, self).init(*args, **kwargs)

  @override
  def reinit(self):
    if not hasattr(self._tls, 'conn'):
      super(DDBPlurksMap, self).reinit()
    self._tls.ddb_plurks_map = DDBTable(config.DYNAMODB_TABLE_PLURKS_MAP, connection=self._tls.conn)

  def get_shard_id(self, plurk_id, strong_consistent=False):
    if not hasattr(self._tls, 'ddb_plurks_map'):
      self.reinit()
    # other things

  def get_shard_ids(self, plurk_ids, strong_consistent=False):
    if not hasattr(self._tls, 'ddb_plurks_map'):
      self.reinit()     
    # other things

  def delete_plurk_id(self, plurk_id):
    if not hasattr(self._tls, 'ddb_plurks_map'):
      self.reinit()
    # other things

  def set_shard_id(self, plurk_id, shard_id):
    if not hasattr(self._tls, 'ddb_plurks_map'):
      self.reinit()
    # other things

This would work, of course, but looking at the duplicated code (even after being converted into a decorator, which is omitted here) made me think of Dr. Chuen-Liang Chen, professor of my design pattern class back in school, who would definitely graded the code with an F.

Well, I would grade my code with an F too, so the quest for a better solution continued…

 
Subclassed Thread-Local Storage to the Rescue

At this point, I knew what I needed — a thread-local storage that could automagically initialize its contents when they were requested for the first time (or, when the thread was first spawned — but as I couldn’t modify thread spawning, this was a no-go). Fortunately, Python’s threading.local does provide such construct: a threading.local instance’s __init__() is called if its thread-local attribute dict, a threading.local-internal data structure, has not been constructed when someone accesses the attribute dict. Such calling of __init__() can be long after the threading.local instance is apparently created in the source code, taking place only when new threads are spawned and those threads’ threading.local instance’s attributes accessed. Now, there’s the big addendum: overriding __init__() in subclasses of threading.local will also be called.

Therefore, my auto-magic was actually simple:

class _DDBBase(Singleton):
  class _TLS(threading.local):
    def __init__(self):
      self.conn = boto.dynamodb2.connect_to_region(
        config.DYNAMODB_REGION,
        aws_access_key_id=config.DYNAMODB_ACCESS_ID,
        aws_secret_access_key=config.DYNAMODB_ACCESS_KEY
      )

  @override
  def init(self, *args, **kwargs):
    self._tls = _DDBBase._TLS()

  def get_api_ver(self):
    return self._tls.conn.APIVersion # automagic!

There, whenever I called _DDBBase().get_api_ver(), the conn attribute would be looked up in self._tls. If at this moment, the thread-local attribute dict had not been constructed as per threading.local‘s internal implementation, _DDBBase._TLS.__init__() would be called upon that self._tls — and this would properly construct the conn attribute and tuck it into the thread-local storage. This would happen once (and only once) per thread, and only when actually needed, so it’s all perfect.

Now, as DDBPlurksMap also needed to store its own data into the thread-local storage, I needed to be able to customize _TLS.__init__()‘s behavior too. Here’s the complete piece of codes:

class _DDBBase(Singleton):
  @hidable
  class _TLS(threading.local):
    def __init__(self):
      self.conn = boto.dynamodb2.connect_to_region(
        config.DYNAMODB_REGION,
        aws_access_key_id=config.DYNAMODB_ACCESS_ID,
        aws_secret_access_key=config.DYNAMODB_ACCESS_KEY
      )

  @override
  def init(self, *args, **kwargs):
    self._tls = self.__class__._TLS()

class DDBPlurksMap(_DDBBase):
  @hide
  class _TLS(_DDBBase._TLS):
    def __init__(self, *args, **kwargs):
      super(DDBPlurksMap._TLS, self).__init__(*args, **kwargs)
      self.ddb_plurks_map = DDBTable(config.DYNAMODB_TABLE_PLURKS_MAP, connection=self.conn)

  def get_shard_id(self, plurk_id, strong_consistent=False):
    try:
      item = self._tls.ddb_plurks_map.get_item(consistent=strong_consistent, id=plurk_id)
    except ItemNotFound:
      return None
    else:
      return int(item['shard_id'])

  def get_shard_ids(self, plurk_ids, strong_consistent=False):
    result_set = self._tls.ddb_plurks_map.batch_get(consistent=strong_consistent, keys=[{'id': plurk_id} for plurk_id in set(plurk_ids)])
    # other things

  def delete_plurk_id(self, plurk_id):
    self._tls.ddb_plurks_map.delete_item(id=plurk_id)

  def set_shard_id(self, plurk_id, shard_id):
    self._tls.ddb_plurks_map.put_item(data={'id': plurk_id, 'shard_id': shard_id}, overwrite=True)

# nothing really to customize, so it's pretty clean for this module
class DDBAtomicCounters(_DDBBase):
  def _inc_and_get(self, key_name):
    res = self._tls.conn.update_item(config.DYNAMODB_TABLE_ATOMIC_COUNTERS, {'name': {'S': key_name}}, {'number': {'Action': 'ADD', 'Value': {'N': '1'}}}, return_values='UPDATED_NEW')
    return int(res['Attributes']['number']['N'])

  def next_magic_id(self):
    return self._inc_and_get('magic_counter')

Well, things are again getting metaclassy here, so let me do a quick step-by-step on what would happen if we call DDBPlurksMap().set_shard_id(1, 2) for example:

  1. For the DDBPlurksMap() part, Singleton.__new__() is called on the DDBPlurksMap class.

    • If an instance of DDBPlurksMap has ever been constructed in the whole Python process, nothing really happens except for us getting that instance.
    • Otherwise, the instance is constructed, and since DDBPlurksMap has no init() defined, _DDBBase.init() is called upon that instance, which creates an instance of our subclassed thread-local storage. Note that self.__class__ is DDBPlurksMap, so we’ll be properly constructing an instance of DDBPlurksMap._TLS, instead of that of _DDBBase._TLS.
  2. Now, inside set_shard_id(), the ddb_plurks_map attribute is requested from self._tls.

    • If at this moment, the subclassed thread-local storage for this thread has been constructed, which means DDBPlurksMap._TLS.__init__() has been called, ddb_plurks_map will be a dynamodb2.table instance, local to the current thread, parameterized by a DynamoDBConnection instance (i.e. conn), also local to the current thread. These two instances would have been constructed within DDBPlurksMap._TLS.__init__() and _DDBBase._TLS.__init__(), respectively (in the reversed order for sure).
    • If not, then threading.local‘s internal implementation calls DDBPlurksMap._TLS.__init__(), which calls _DDBBase._TLS.__init__() to create and store a DynamoDBConnection instance into the thread-local storage. And then, DDBPlurksMap._TLS.__init__() creates and store a dynamodb2.table instance into the thread-local storage. Those instances are only visible to the current thread, no racing!
  3. Either way, a thread-local dynamodb2.table instance will be returned by the __getattr__() call on the thread-local storage for the ddb_plrurks_map attribute, and everything is good! Not many duplicated codes!

 
Afterwords
Really, as Greg Weng, my former colleague at Mozilla, puts it, threads might be too heavy a weapon to maximize request throughput by working around I/O blockage. In the future, we should really begin investigating into those state-of-the-art async I/O constructs such as coroutines and futures and the like. Too bad we’re still stuck in Python 2.7, though…

Read More

In the Making of Mnjul’s Intimate Home 4931 (XI)

上一篇在這裡喔。

總之就是想到什麼可以改進的地方,或是看到什麼新技術覺得可以拿來學學用,就來試試看~

  • 關於各種用 XHR 載入的 binary resource ,像是字型跟圖片,其實不用轉成 Data URL,直接用 Blob URL 就好啦!畢竟現在都 2016 年了。這樣一來,其實發 XHR 的時候也不需要使用 arraybuffer 的回傳型態,直接指定 blob 回傳型態,然後一口氣使用 URL.createObjectURL() 就好。當然也不用 arraybuffer to base64 converter。不過這樣一來,我的 loadfile.php 就要正確地吐 Content-Type header 就是,不過這也是小事一樁啦,畢竟檔案都是我的。至少 inspector 的 DOM 帳面上看起來比較乾淨,像是 style tag 裡面就不會有一大沱的 base64 字型檔案內容。
  • 本來一直想要把我自己寫的 jQuery.fn.crossCSSByValue 這個 helper function ( 用來在 css value,而不是 property,加上 vendor prefix )改用 jQuery cssHooks 來做,但思索半天都覺得不會比較簡單,就放棄了。
  • Windows 10 的 Edge 瀏覽器可以正確顯示我自己亂搞過的華文仿宋 web font,所以我決定一口氣不支援 IE 11,也不修 IE 11 上面的各種 bug了。
  • 在 Fragments 或 Shards 慢慢載入內容的時候,我之前不知道為什麼寫了 $parentContainer.append($child[0].outerHTML); 這種 code。實際上只要寫 $parentContainer.append($child); 就好了啊啊啊啊,省了 serialization to HTML 跟 重新 parse 的時間!
  • Ego、Fragments、Shards 的滑鼠 hover 會變亮的 code,其實除了正常的 :hover CSS 以外,還有「滑鼠滾輪捲動時,如果游標在文字上面,也要自動變亮」的 code。以前是用 document.elementFromPoint(),然後再用一個 while loop 一路往 DOM tree parent 找來決定滑鼠是不是在一組文字架構上,然後正確打亮那整組文字架構。後來發現 jQuery 的 closest() 就可以做到這個功能,而且整體 code 看起來比較簡單。
    • 我原來的 while loop 其實不用一直 backtrack 到 <body> 才知道找失敗,而是到某個 container element 就知道找失敗了,可以 early break。
    • 但 break 的時候必須要看有沒有找到,就要額外設一個 flag ,然後 loop 外面再針對 flag 做事。
    • 但因為 closest() 也可以傳入多個 selector,所以同樣的邏輯只要看 closest() 的回傳值就可以搞定了。
  • 接受了太太的建議,把 Ubuntu 字型換成 DIN 字型,比較沉穩一點。另外也改正了 @font-face 的用法,把 font-weight 正確放在同一個 font-family 的不同的 @font-face 裡面。
  • 背景音樂拿掉 OGG 格式啦!因為 Firefox 有陣子前就開始不管是什麼作業系統都支援 MP3 了。
  • 另外,CSS 跟 JS minification 改用 clean-css 跟 UglifyJS 了( 依然採用 Refresh-SF 的線上服務,我好懶得自己做 automation tool 啊!)
    • 但是 clean-css 也會把 hsla(--var(-something), 0%, xx%, xx) 弄成 hsla(--var(-something),0,xx%,xx),那個 saturation 少了百分比,Chrome 就爛掉了,我還得手動用 regexp 改回來。

Read More

OS X El Capitan: Displace ~/Library/Caches to Another Disk

Prologue

So, to save money and real estate, here in NYC I have been using a quad-core Late-2012 Mac Mini, which was the cheapest (and reasonably new) quad-core Mac still floating around last year, with aftermarket 16GB RAM upgrades, and the change from the pitiful 5400RPM hard disk to a Samsung 1TB SSHD with 8GB NAND cache. I didn’t want to place all my data on an external disk — especially when it’s connected through USB, so I wouldn’t want to only have one SSD inside the Mac — even though I knew my usage pattern, along with OS X’s memory management strategy, would incur lots of swap usage, I couldn’t afford a 128GB SSD + 1TB HDD setup. So I gave a shot with the SSHD.

Well, time went by and it turned out that while the machine did a wonderful job when I was editing my dozens-layers HeHuan Mountain star trails photo, or transcoding videos, it actually performed like hell during day-to-day usage, when I switched programs (do I need to say “apps” here?) frequently and, taking browsers as example, when data (transient or permanent) was read from and written to disk in super small chunks. And I always recorded a swap usage at around 8GB to 12GB, and the SSHD was falling short of tempering my impatience.

So earlier this year I began to ponder on getting an SSD for special purposes. Unexpectedly, after a year, SSD price has dropped significantly, and I could now get a 250GB Samsung SSD for less than $80 before tax (it’s way cheaper than the aftermarket vendor’s offer). Though, that’s a TLC one! But I knew what my special purpose was going to be: virtual machines (only OS and programs, not data), browser caches for Firefox and Chrome, and Adobe Camera Raw caches, and currently-being-processed photos. These were more or less not mission-critical, and I still employed daily rsync backups to my NAS, so any data loss due to it being a TLC SSD and USB-connected would be bearable.

Oh and, while I could have bought an iFixit toolkit to hack the SSD into the Mac (which also gave me TRIM support), I forsook that and only used a USB to SATA converter. While the latter did have drawbacks like USB protocol overheads and the CPU processing power needed to handle the USB stack, the overall access latency (and even bandwidth) was still significantly improved.

The Actual Thing (of this post’s title)

But that’s actually quite not enough. Even though I could put Firefox and Chrome cache to the SSD (the latter was actually being done through symlinks, and the former using about:config), there were still tons of programs and user-related system services that accessed ~/Library/Caches which were not benefiting from the SSD. While I thought it could easily be mitigated by symlinking the whole ~/Library/Caches off-disk, this soon proved to be wrong.

While the system didn’t crash (even after reboots) and programs didn’t behave obviously erratically, I soon discovered that my Notes.app (okay, “app”) was not syncing with iCloud. Further investigations in Console.app showed that… (only showing relevant stack frames here for brevity)

May 15 16:16:19 mnjulnimi kernel[0]: Sandbox: cloudd(10110) deny(1) file-write-create /Volumes/Cache_VM/MnjulLibraryCaches/CloudKit
May 15 16:16:19 mnjulnimi cloudd[10110]: *** Terminating app due to uncaught exception 'NSGenericException', reason: 'Error opening db at /Users/USER/Library/Caches/*/CloudKitMetadata: <CKError 0x7f9602433850: "Internal Error" (1); "Error creating db dir at /Users/USER/Library/Caches/CloudKit: Error Domain=NSCocoaErrorDomain Code=513 "You don't have permission to save the file "CloudKit" in the folder "Caches"." UserInfo={NSFilePath=/Users/mnjul/Library/Caches/CloudKit, NSUnderlyingError=0x7f960270b180 {Error Domain=NSPOSIXErrorDomain Code=1 "Operation not permitted"}}">'
        *** First throw call stack:
        (
                2   CoreFoundation                      0x00007fff947a84bd +[NSException raise:format:] + 205
                3   CloudKit                            0x00007fff8d902d8c +[CKSQLiteError raise:code:extended:] + 1660
                4   CloudKit                            0x00007fff8d8ff148 -[CKSQLite raise:] + 1055
                5   CloudKit                            0x00007fff8d8fe82b -[CKSQLite open] + 145
                6   CloudKitDaemon                      0x00007fff932f41cf -[CKDMetadataCache init] + 262
        )
May 15 16:16:19 mnjulnimi com.apple.xpc.launchd[1] (com.apple.cloudd[10110]): Service exited due to signal: Abort trap: 6

Yikes. cloudd’s CloudKit library’s SQLite had no permission to create a file (or directory for that matter) in my symlinked-to caches folder. At first sight I thought it was the usual file mode/owner/ACL issues, but after some mangling around with chmod/chown, I found everything was actually already good. It’s at this moment I saw cloudd was access-denied from sandboxd. Okay. So, OS X was trying to restrict placement of (potentially sensitive) user data into external file systems. This actually made sense, but, well, I knew what I was doing. Googling around using a mix of keywords brought me to an answer from Apple Support Communities that instructed whitelisting the symlinked-to directory in sandboxd profile.

Here’s the detour: OS X El Capitan has System Integrity Protection, where even root user cannot modify the said application.sb file when the system is booted normally, and for that, I had to reboot into Recovery to modify the file. And a double detour: vim in that Recovery was somehow… the plain vi. Yes. I needed to use hjkl for navigation, and x for deleting chars. Man I was proud for being old school.

Well, it turned out the detours weren’t worthwhile, as the whitelisting seemed to have no effect (according to the same discussion thread, it actually apparently ceased to have any effect since Mavericks). So it appeared that I had to ditch symlink and go down a more convoluted/low-level way.

From my OS knowledge, I couldn’t use hard links since I was crossing file system boundaries. This left me only one choice to experiment with: split out a new partition inside my SSD that would only be used for the ~/Library/Caches, and mount it directly on the directory.

It sounds nasty but it’s actually not as hard to do. The SSD (i.e. the original single partition that’s as large as the disk) was not really full and was already on GPT, with the existing partition on HFS+, so resizing/appending partitions without data loss wasn’t hard (but better do that in Recovery’s Disk Utilities though, not on a live OS). And then, since this mounting had to be ready before I logged in, I needed (only) to modify /etc/fstab — never thought I’d touch this file on OS X, but it’s good enough that BSD fstabs were pretty much in the same format as Linux ones, which I’d be more familiar with. Upon some extra googling, using UUID to specify the device was said commendable over using file paths, so I needed to dig out the UUID using diskutil info. From my experience, I needed to use Volumn UUID, instead of Disk / Partition UUID. Reboot — Voila, Notes.app was syncing, no issue in Console.app. (I didn’t need to copy the old cache files into the new partition, since they were, well, caches.)

Epilogue

In the end, the configuration was good (as far as performance is concerned, I didn’t do a formal benchmark so I couldn’t tell if it’s effect-wise a placebo) until I was updating my iPhone’s iOS from 9.3.1 to 9.3.2 using iTunes: during the process, iTunes needed to use the caches directory to store something that was nearly 2GB large. But I had allocated only 2GB to the cache partition (I had never observed it exceeding 1GB, but guess this kind of usage spikes happen), so it failed due to insufficient disk space. So I went to Recovery again, and adjusted the partition to be 10GB large. (I also updated OS X to 10.11.5 afterwards, which went without a hiccup). But that also means I have almost 8~9GB of precious SSD space wasted. Uhm. I’ll probably convince myself that it allows the SSD to do GC and wear-leveling more nicely, since I don’t have TRIM over USB. But that anyway depends on how Samsung’s MGX controller is doing things.

For sure, this is still far from hacking for optimal situations: I can even try to hack things to put swap files in SSD (probably following a detailed step-by-step posted in StackExchange), or put /Library/Caches or /System/Library/Caches there, but that’s gonna need some careful planning (the latter two directories don’t have large volumes of file as far as I can see, though). Additionally, I’m also wondering if I can put the whole Spotlight (mds) database of my SSHD into the SSD with the same trick, or whatsoever.

Read More

Typesetting Resumes with HTML/CSS

The other day my wife was busy redesigning her resume as her graduation was imminent. And I decided to completely redo my resume too — in a geeky way. I looked at my wife’s resume template (in Adobe InDesign), and contemplated the possibility if I could actually reproduce it with plain HTML and CSS (not even JavaScript and SVG!), if I allowed sparing some pixel-perfect precision.

It turned out to be not that hard: because everything was to be contained in a fix-sized box (e.g. the 8″-by-11.5″ paper), ordinary CSS plus the @page at-rule produced correctly laid-out results, both on screen and printed-to-PDF. With some CSS variables and my prior front-end experiences, I could even adapt the HTML/CSS structure to arbitrary resume contents; even the resume contained some lines-and-shapes decorations, I didn’t have to employ really dirty hacks to get things to look nice.

So, I decided to formally created a GitHub repository with it: html-resume.

Oh, and here’s the actual resume. I must admit I got a little bit guilty when I wrote “no proprietary software required” in the repo readme: I still had to open Adobe Acrobat to add hyperlinks to my contact info in the online PDF.

Well, arguably, even though I tried to minimize dirty hacks, I can’t say I didn’t mess things up — I believe there are still unnecessary margin/padding or positional rules, and there are strange magic numbers popping up just to align things (even though the use of CSS variables did help hide some). Take as example the icons and the vertical lines protruding from them — they have zero robustness, and are likely to break if you’re to adjust the spacing, or the line width, or the icon size.

It’s also my first attempt to (almost) solely use inches and points in CSS, and I still don’t really get the idea when I should use inches and when points. For spacing and positioning like margins or padding, using inches seem reasonable, while for font (and font icon) sizes, points are the way to go (since I’m typesetting, right?) — what if I need to make a space that’s combinatively deriving from another spacing defined by inches and, say, an icon? While CSS calc() hides the issue for me, I still have not reconciled on it. Also, I still need to employ pixel units from time to time, to get rid of misalignments.

Anyway, it’s a nice fun pet project for me. Next round is to change my CV webpage!

Epilogue for anyone looking for talented UX designers: my wife is super diligent and smart, who will be graduating from Parsons MFA Design & Technology this summer. She already holds a degree in computer science and also has solid background in graphic designs. She’s likely the very UX designer candidate you’ve been searching for — checkout her (current) resume and (a bit dated) portfolio!

Read More

In the Making of Mnjul’s Intimate Home 4931 (X)

上一篇在這裡喔。

最近又有點時間,就把網站的上次還沒做完的,比較耗工的改動弄完。

  • 增進了 Fragments 跟 Shards loading 的流暢度( 但整體速度是下降的 )。我之前的作法是,算瀏覽器每次給我 requestAnimationFrame 時隔多久,就用那個「多久」來決定我一次可以花多少時間在 loading 上面( 然後就用 setTimeout 把掌控權還給瀏覽器 )。想想不對啊,如果這個時間隔很久,代表瀏覽器很忙,沒辦法讓 client js 一直畫圖,那不就代表我也不應該花很久的時間來load 那些 DOM elements?所以就需要用新的辦法。結果,跟以前 Mozilla 的同事討論了以後,我發現我要做的事情根本是「在 cooperative multitasking OS 下面,有一個 process 想要吃掉所有的 CPU,又不想讓別的 process 發現變慢;但又拿不到別的 process 到底會多吃 CPU」--所以我懶得去想夠棒的 heuristic,就隨便設了一些 timeout 了。
  • 最近發現 Firefox 也支援了 -moz-osx-font-smoothing,用 grayscale 值,可以達到跟 -webkit-font-smoothing: subpixel-antialiased 一樣,在黑底白字時文字粗細比較細的效果。萬!歲!
  • 動態載入 webfont 的 code,會把 encode 好的 base64 webfont 取代進我 CSS 的某個字串;本來這個字串就只是一行在 rule block 裡面像是 UBUNTU_FONT 的字串,不過因為這樣 CSS syntax check 會爆掉,所以現在改用比較搞剛的像是:-mih-webfont: ubuntu;
  • 至於為什麼會有 CSS syntax check?因為我也開始用 css linter 跟 js linter 來讓自己的東西變得更 robust 啦。當然我都是 lint 還沒有 minify 的結果。
  • 所有的 id 跟 class identifier naming convention 也採用主流的 lowercase-delimited-by-hyphen。
  • 另外,重寫了Arraybuffer to base64 converter。
  • 也再度最佳化 jQuery selector 使用,並且減少無意義的 class。像是 Fragments 有兩百個項目,不用每個項目都掛 fragments-item class,而用 parent > child selector 選。
  • text-shadow 現在有 transition 了。

Read More

New Personal Homepage!

So it’s been five years I last made a new version of mnjul.net and almost two years after I took the homepage down. Since I have been relatively having more free time, I’ve just launched a brand new personal homepage embodied with my recent front-end knowledges. Of the notable are:

  • I’m finally making some responsive-web-design by not serving two different templates — the viewport currently supports width down to 1000px (with default is 1200px) to be tablet friendly, done with media queries. Well, I haven’t decided how smartphone viewport should look though.
  • CSS filter effects for monochroming thumbnails and giving back full colors when hover. Comes from CSS Secrets by Lea Verou which by the way is a very nice read.
  • A rippling effect on button hover, purely implemented with CSS
  • The wavy section junctions are dynamically drawn with canvas, and on top of that, resizes nicely when the viewport resizes.
  • I have finally gave in and do the LESS compilation on my local machine as one of my deployment step (automated with Sublime Text packages for that matter). The huge benefit for me is that I can combine the LESS and normalize.css and Font Awesome in one (big) CSS file to reduce network overhead.

For now, I’m still considering the homepage as beta. Forthcoming features include a smartphone-screen friendly viewport plus some layout optimization for desktop/tablet viewport, a PHP-backed contact me feature, and some other minor visual/UI tweaks. But needless to say I’m super satisfied with what I’ve done as somebody with no UX/UI/visual design taste :p. On top of this, let me thank my wife, Angela, for giving me valuable suggestions and inspirations regarding UX/UI/visual designs.

Besides mnjul.net, I have snatched other domains that relate to my name. For the homepage served through those other domains, I have enabled Cloudfront CDN. My ultimate goal is to server mnjul.net on CDN too (but since mnjul.net has some other webpages and I’ll eventually hook up a contact me feature, this still needs some planning). Additionally, I currently always serve 2x (“retina”) version of thumbnails sprites — should be serving 1x version if the browser is not HiDPI or retina.

Read More

Wheel Reinvention: A JavaScript Webfont Loader, with ProgressEvent

So I made such a GitHub project as extracted from Allodynia.

A quick excerpt: Even though there are a heck lot of JavaScript or CSS Font Loading Module-based web font loaders, most of them don’t have progress event. I wanted such event so bad (for loading CJK fonts), I had to make my own. For such propose, the loader uses naïve XMLHTTPRequest to load a webfont file from the remote server, and injects such webfont into CSS in the form of Data URI. The progress event is exposed by the XHR.

Read More

Migrating Ubuntu AWS EC2 t1.micro instance to t2.micro

Alright. I decided to write this because the tutorials I could find on the internet weren’t really tailored specifically for Ubuntu and it turned out that extra care might be needed.

So as we all know, t1 instances are paravirtualization and t2 are hardware virtual machines, and naïvely re-attching Linux root device EBS created for t1 instances onto new t2 instance as-is wouldn’t work due to potential boot loader and kernel issues — we need some manual operations.

My Ubuntu installation was a 32-bit 14.04.2 server with all updates applied. I installed 32-bit back in 2012 because at that time t1.micro’s less than 700MB available RAM didn’t seem enough to warrant 64-bit for me. It’s not possible to “upgrade” (if that’s a suitable word at all) the installation to 64-bit in-place so my resulting t2.micro installation would still be 32-bit (anyway it’s just 1GB RAM). Also, I had already had grub installed (not sure if that’s default, though).

Anyway, I mostly followed Javier Sianes‘s steps. His steps are for Amazon Linux instances, and in adaptation to Ubuntu instances there are some things to note:

  • Choose Amazon Linux AMI when launching the working instance, even though we’re migrating Ubuntu instances. When I launched Ubuntu as my working instance and attached the two (t1 & t2) EBS volumes onto it, the working instance booted on the t2 volume (located at /dev/xvdg), instead of its root device attached at /dev/sda1 (as shown in EC2 console). I couldn’t understand why this happened as everything indicated normal at EC2 console. I guess you could launch an Ubuntu working instance and attach the t1 & t2 volumes only after the working instance boots up, but I didn’t try.
  • I chose 12GB EBS storage when launching the t2 instance to match my t1 instance storage space. However the Ubuntu AMI’s root partition/filesystem was only 8GB large. I needed to re-create the partition table and resize the filesystem. Not that hard as I’d did that quite a few times.
  • Remember that the root device node for Ubuntu is /dev/sda1.
  • Replacing the /boot and kernel folders was necessary for first boot after the migration of the files. I guess it’s because the 64-bit kernel was still needed for that boot. After that, I just manually re-installed the (32-bit) kernel images by apt-get and let apt-get configure boot-related mechanisms.
  • Another thing to note is t2.micro instances must be incorporated in a VPC network. My old Elastic IP didn’t seem to be a VPC-capable public IP (in 175.41.224.0/19, in Tokyo) so I had to redo all the binding and DNS setup.

Read More