[OpenIndiana-discuss] ZFS read speed(iSCSI)
Heinrich van Riel
heinrich.vanriel at gmail.com
Sat Jun 8 22:57:06 UTC 2013
I took a look at every server that I knew I could power down or that is
slated for removal in the future and I found a qlogic adapter not in use.
HBA Port WWN: 2100001b3280b
Port Mode: Target
Port ID: 12000
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: QLE2460
Firmware Version: 5.2.1
FCode/BIOS Version: N/A
Serial Number: not available
Driver Name: COMSTAR QLT
Driver Version: 20100505-1.05
Type: F-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b3280b
Link does not go down but useless, right from the start it is as slow as
the emulex after I made the xfer change.
So it is not a driver issue.
alloc free read write read write
----- ----- ----- ----- ----- -----
681G 53.8T 5 12 29.9K 51.3K
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 88 0 221K
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 163 0 812K
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 198 0 1.13M
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 88 0 221K
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 187 0 1.02M
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
681G 53.8T 0 0 0 0
This is a clean install of a7 with nothing done other than nic config in
lacp. I did not attempt a reinstall of a5 yet and prob won't either.
I dont know what to do anymore I was going to try OmniOS but there is no
way of knowing if it would work.
I will see if I can get approved for a solaris license for one year, if not
I am switching back to windows storage spaces. Cant backup the current lab
on the EMC array to this node in any event since there is no ip
connectivity and fc is a dream.
Guess I am the only one trying to use it as an fc target and these problems
are not noticed.
On Sat, Jun 8, 2013 at 4:55 PM, Heinrich van Riel <
heinrich.vanriel at gmail.com> wrote:
> changing max-xfer-size causes the link to stay up and no problem are
> reported from stmf.
>
> # Memory_model max-xfer-size
> # ----------------------------------------
> # Small 131072 - 339968
> # Medium 339969 - 688128
> # Large 688129 - 1388544
> #
> # Range: Min:131072 Max:1388544 Default:339968
> #
> max-xfer-size=339968;
>
> as soon as I changed it to 339969 the there is no link loss, but I would
> be so lucky that is solves my problem. after a few min it would grind to a
> crawl, so much so that in vmware it will take well over a min to just
> browse a folder, we talking are a few k/s.
>
> Setting it to the max causes the the link to go down again and smtf
> reports the following again:
> FROM STMF:0062568: abort_task_offline called for LPORT: lport abort timed
> out
>
> I also played around with the buffer settings.
>
> Any ideas?
> Thanks,
>
>
>
> On Fri, Jun 7, 2013 at 8:38 PM, Heinrich van Riel <
> heinrich.vanriel at gmail.com> wrote:
>
>> New card, different PCI-E slot (removed the other one) different FC
>> switch (same model with same code) older hba firmware (2.72a2) = same
>> result.
>>
>> On the setting changes when it boots it complains about this option, does
>> not exist: szfs_txg_synctime
>> The changes still allowed for a constant write, but at a max of 100Mb/s
>> so not much better than iscsi over 1Gbe. I guess I would need to increase
>> write_limit_override. if i disable the settings again it shows 240MB/s
>> with bursts up to 300, both stats are from VMware's disk perf monitoring
>> while cloning the same VM.
>>
>> All iSCSI luns remain active with no impact.
>> So I will conclude, I guess, it seems to be the problem that was there in
>> 2009 from build ~100 to 128. When I search the error messages all posts
>> date back to 2009.
>>
>> I will try one more thing to reinstall with 151a5 since a server that was
>> removed from the env was running this with no issues, but was using an
>> older emulex HBA, LP10000 PCIX
>> Looking at the notable changes in the release notes past a5 I do see
>> anything that changed that I would think would cause the behavior. Would
>> this just be a waste of time?
>>
>>
>>
>> On Fri, Jun 7, 2013 at 6:36 PM, Heinrich van Riel <
>> heinrich.vanriel at gmail.com> wrote:
>>
>>> In the debug info I see 1000's of the following events:
>>>
>>> FROM STMF:0149225: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149225: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149225: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149226: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149226: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149226: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149227: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149227: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149227: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> emlxs1:0149228: port state change from 11 to 11
>>> FROM STMF:0149228: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149228: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149228: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> :0149228: fct_port_shutdown: port-ffffff1157ff1278, fct_process_logo:
>>> unable to
>>> clean up I/O. iport-ffffff1157ff1378, icmd-ffffff1195463110
>>> FROM STMF:0149229: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149229: abort_task_offline called for LPORT: lport abort
>>> timed out
>>> FROM STMF:0149229: abort_task_offline called for LPORT: lport abort
>>> timed out
>>>
>>>
>>> And then the following as the port recovers.
>>>
>>> emlxs1:0150128: port state change from 11 to 11
>>> emlxs1:0150128: port state change from 11 to 0
>>> emlxs1:0150128: port state change from 0 to 11
>>> emlxs1:0150128: port state change from 11 to 0
>>> :0150850: fct_port_initialize: port-ffffff1157ff1278, emlxs initialize
>>> emlxs1:0150950: port state change from 0 to e
>>> emlxs1:0150953: Posting sol ELS 3 (PLOGI) rp_id=fffffd lp_id=22000
>>> emlxs1:0150953: Processing sol ELS 3 (PLOGI) rp_id=fffffd
>>> emlxs1:0150953: Sol ELS 3 (PLOGI) completed with status 0, did/fffffd
>>> emlxs1:0150953: Posting sol ELS 62 (SCR) rp_id=fffffd lp_id=22000
>>> emlxs1:0150953: Processing sol ELS 62 (SCR) rp_id=fffffd
>>> emlxs1:0150953: Sol ELS 62 (SCR) completed with status 0, did/fffffd
>>> emlxs1:0151053: Posting sol ELS 3 (PLOGI) rp_id=fffffc lp_id=22000
>>> emlxs1:0151053: Processing sol ELS 3 (PLOGI) rp_id=fffffc
>>> emlxs1:0151053: Sol ELS 3 (PLOGI) completed with status 0, did/fffffc
>>> emlxs1:0151054: Posting unsol ELS 3 (PLOGI) rp_id=fffc02 lp_id=22000
>>> emlxs1:0151054: Processing unsol ELS 3 (PLOGI) rp_id=fffc02
>>> emlxs1:0151054: Posting unsol ELS 20 (PRLI) rp_id=fffc02 lp_id=22000
>>> emlxs1:0151054: Processing unsol ELS 20 (PRLI) rp_id=fffc02
>>> emlxs1:0151055: Posting unsol ELS 5 (LOGO) rp_id=fffc02 lp_id=22000
>>> emlxs1:0151055: Processing unsol ELS 5 (LOGO) rp_id=fffc02
>>> emlxs1:0151146: Posting unsol ELS 3 (PLOGI) rp_id=21500 lp_id=22000
>>> emlxs1:0151146: Processing unsol ELS 3 (PLOGI) rp_id=21500
>>> emlxs1:0151146: Posting unsol ELS 20 (PRLI) rp_id=21500 lp_id=22000
>>> emlxs1:0151146: Processing unsol ELS 20 (PRLI) rp_id=21500
>>> emlxs1:0151146: Posting unsol ELS 3 (PLOGI) rp_id=21600 lp_id=22000
>>> emlxs1:0151146: Processing unsol ELS 3 (PLOGI) rp_id=21600
>>> emlxs1:0151146: Posting unsol ELS 20 (PRLI) rp_id=21600 lp_id=22000
>>> emlxs1:0151146: Processing unsol ELS 20 (PRLI) rp_id=21600
>>> emlxs1:0151338: Posting unsol ELS 3 (PLOGI) rp_id=21500 lp_id=22000
>>> emlxs1:0151338: Processing unsol ELS 3 (PLOGI) rp_id=21500
>>> emlxs1:0151338: Posting unsol ELS 20 (PRLI) rp_id=21500 lp_id=22000
>>> emlxs1:0151338: Processing unsol ELS 20 (PRLI) rp_id=21500
>>> emlxs1:0151338: Posting unsol ELS 3 (PLOGI) rp_id=21600 lp_id=22000
>>> emlxs1:0151338: Processing unsol ELS 3 (PLOGI) rp_id=21600
>>> emlxs1:0151338: Posting unsol ELS 20 (PRLI) rp_id=21600 lp_id=22000
>>> emlxs1:0151338: Processing unsol ELS 20 (PRLI) rp_id=21600
>>> emlxs1:0151428: Posting unsol ELS 3 (PLOGI) rp_id=21500 lp_id=22000
>>> emlxs1:0151428: Processing unsol ELS 3 (PLOGI) rp_id=21500
>>> emlxs1:0151428: port state change from e to 4
>>> emlxs1:0151428: Posting unsol ELS 20 (PRLI) rp_id=21500 lp_id=22000
>>> emlxs1:0151428: Processing unsol ELS 20 (PRLI) rp_id=21500
>>> emlxs1:0151428: Posting unsol ELS 3 (PLOGI) rp_id=21600 lp_id=22000
>>> emlxs1:0151428: Processing unsol ELS 3 (PLOGI) rp_id=21600
>>> emlxs1:0151428: Posting unsol ELS 20 (PRLI) rp_id=21600 lp_id=22000
>>> emlxs1:0151428: Processing unsol ELS 20 (PRLI) rp_id=21600
>>>
>>> To be honest it does not really tell me much since I do not understand
>>> comstar to these depths. It would appear that the link fails so either
>>> driver problem or hardware issue? I will replace the LPe11002 with a brand
>>> new unopened one and then give up on FC on OI.
>>>
>>>
>>>
>>>
>>> On Fri, Jun 7, 2013 at 4:54 PM, Heinrich van Riel <
>>> heinrich.vanriel at gmail.com> wrote:
>>>
>>>> I did find this in my inbox from 2009, I have been using FC with ZFS
>>>> for quite sometime and only recently retired an install with OI a5 that was
>>>> upgraded from opensolaris. It did not do real heavy duty stuff, but I had a
>>>> similar problem where we were stuck on build 99 for quite some time.
>>>>
>>>> To Jean-Yves Chevallier at emulex
>>>> Any comments on the future of Emulex with regards to the COMSTAR
>>>> project?
>>>> It seems I am not the only one that have problems using Emulex in later
>>>> builds. For now I am stuck with build 99.
>>>> As always any feedback would be greatly appreciated since we have to
>>>> make a decision of sticking with Opensolaris & COMSTAR or start migrating
>>>> to another solution since we cannot stay on build 99 forever.
>>>> What I am really trying to find out is if there is a roadmap/decision
>>>> to ultimately only support Qlogic HBA’s in target mode.
>>>>
>>>> Response:
>>>>
>>>>
>>>> Sorry for the delay in answering you. I do have news for you.
>>>> First off, the interface used by COMSTAR has changed in recent Nevada
>>>> releases (NV120 and up I believe). Since it is not a public interface we
>>>> had no prior indication on this.
>>>> We know of a number of issues, some on our driver, some on the COMSTAR
>>>> stack. Based on the information we have from you and other community
>>>> members, we have addressed all these issues in our next driver version – we
>>>> will know for sure after we run our DVT (driver verification testing) next
>>>> week. Depending on progress, this driver will be part of NV 128 or else NV
>>>> 130.
>>>> I believe it is worth taking another look based on these upcoming
>>>> builds, which I imagine might also include fixes to the rest of the COMSTAR
>>>> stack.
>>>>
>>>> Best regards.
>>>>
>>>>
>>>> I can confirm that this was fixed in 128 and all I did was update from
>>>> 99 to 128 and there were no problems.
>>>> Seem like the same problem has now returned and emulex does not appear
>>>> to be a good fit since sun mostly used qlogic.
>>>>
>>>> guess it is back to iscsi only for now.
>>>>
>>>>
>>>>
>>>> On Fri, Jun 7, 2013 at 4:40 PM, Heinrich van Riel <
>>>> heinrich.vanriel at gmail.com> wrote:
>>>>
>>>>> I changed the settings. I do see it writing all the time now, but the
>>>>> link still dies after a a few min
>>>>>
>>>>> Jun 7 16:30:57 emlxs: [ID 349649 kern.info] [ 5.0608]emlxs1:
>>>>> NOTICE: 730: Link reset. (Disabling link...)
>>>>> Jun 7 16:30:57 emlxs: [ID 349649 kern.info] [ 5.0333]emlxs1: NOTICE:
>>>>> 710: Link down.
>>>>> Jun 7 16:33:16 emlxs: [ID 349649 kern.info] [ 5.055D]emlxs1: NOTICE:
>>>>> 720: Link up. (4Gb, fabric, target)
>>>>> Jun 7 16:33:16 fct: [ID 132490 kern.notice] NOTICE: emlxs1 LINK UP,
>>>>> portid 22000, topology Fabric Pt-to-Pt,speed 4G
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jun 7, 2013 at 3:06 PM, Jim Klimov <jimklimov at cos.ru> wrote:
>>>>>
>>>>>> Comment below
>>>>>>
>>>>>>
>>>>>> On 2013-06-07 20:42, Heinrich van Riel wrote:
>>>>>>
>>>>>>> One sec apart cloning 150GB vm from a datastore on EMC to OI.
>>>>>>>
>>>>>>> alloc free read write read write
>>>>>>> ----- ----- ----- ----- ----- -----
>>>>>>> 309G 54.2T 81 48 452K 1.34M
>>>>>>> 309G 54.2T 0 8.17K 0 258M
>>>>>>> 310G 54.2T 0 16.3K 0 510M
>>>>>>> 310G 54.2T 0 0 0 0
>>>>>>> 310G 54.2T 0 0 0 0
>>>>>>> 310G 54.2T 0 0 0 0
>>>>>>> 310G 54.2T 0 10.1K 0 320M
>>>>>>> 311G 54.2T 0 26.1K 0 820M
>>>>>>> 311G 54.2T 0 0 0 0
>>>>>>> 311G 54.2T 0 0 0 0
>>>>>>> 311G 54.2T 0 0 0 0
>>>>>>> 311G 54.2T 0 10.6K 0 333M
>>>>>>> 313G 54.2T 0 27.4K 0 860M
>>>>>>> 313G 54.2T 0 0 0 0
>>>>>>> 313G 54.2T 0 0 0 0
>>>>>>> 313G 54.2T 0 0 0 0
>>>>>>> 313G 54.2T 0 9.69K 0 305M
>>>>>>> 314G 54.2T 0 10.8K 0 337M
>>>>>>>
>>>>>> ...
>>>>>> Were it not for your complaints about link resets and "unusable"
>>>>>> connections, I'd say this looks like a normal behavior for async
>>>>>> writes: they get cached up, and every 5 sec you have a transaction
>>>>>> group (TXG) sync which flushes the writes from cache to disks.
>>>>>>
>>>>>> In fact, the picture still looks like that, and possibly is the
>>>>>> reason for hiccups.
>>>>>>
>>>>>> The TXG sync may be an IO intensive process, which may block or
>>>>>> delay many other system tasks; previously when the interval
>>>>>> defaulted to 30 sec we got unusable SSH connections and temporarily
>>>>>> "hung" disk requests on the storage server every half a minute when
>>>>>> it was really busy (i.e. initial filling up with data from older
>>>>>> boxes). It cached up about 10 seconds worth of writes, then spewed
>>>>>> them out and could do nothing else. I don't think I ever saw network
>>>>>> connections timing out or NICs reporting resets due to this, but I
>>>>>> wouldn't be surprised if this were the cause for your case, though
>>>>>> (i.e. disk IO threads preempting HBA/NIC threads for too long
>>>>>> somehow, making the driver very puzzled about staleness state of its card).
>>>>>>
>>>>>> At the very least, TXG syncs can be tuned by two knobs: the time
>>>>>> limit (5 sec default) and the size limit (when the cache is "this"
>>>>>> full, begin the sync to disk). The latter is a realistic figure that
>>>>>> can allow you to sync in shorter bursts - with less interruptions
>>>>>> to smooth IO and process work.
>>>>>>
>>>>>> A somewhat related tunable is the number of requests that ZFS would
>>>>>> queue up for a disk. Depending on its NCQ/TCQ abilities and random
>>>>>> IO abilities (HDD vs. SSD), long or short queues may be preferable.
>>>>>> See also: http://www.solarisinternals.**com/wiki/index.php/ZFS_Evil_*
>>>>>> *Tuning_Guide#Device_I.2FO_**Queue_Size_.28I.2FO_**Concurrency.29<http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29>
>>>>>>
>>>>>> These tunables can be set at runtime with "mdb -K", as well as in
>>>>>> the /etc/system file to survive reboots. One of our storage boxes
>>>>>> has these example values in /etc/system:
>>>>>>
>>>>>> *# default: flush txg every 5sec (may be max 30sec, optimize
>>>>>> *# for 5 sec writing)
>>>>>> set zfs:zfs_txg_synctime = 5
>>>>>>
>>>>>> *# Spool to disk when the ZFS cache is 0x18000000 (384Mb) full
>>>>>> set zfs:zfs_write_limit_override = 0x18000000
>>>>>> *# ...for realtime changes use mdb.
>>>>>> *# Example sets 0x18000000 (384Mb, 402653184 b):
>>>>>> *# echo zfs_write_limit_override/**W0t402653184 | mdb -kw
>>>>>>
>>>>>> *# ZFS queue depth per disk
>>>>>> set zfs:zfs_vdev_max_pending = 3
>>>>>>
>>>>>> HTH,
>>>>>> //Jim Klimov
>>>>>>
>>>>>>
>>>>>> ______________________________**_________________
>>>>>> OpenIndiana-discuss mailing list
>>>>>> OpenIndiana-discuss@**openindiana.org<OpenIndiana-discuss at openindiana.org>
>>>>>> http://openindiana.org/**mailman/listinfo/openindiana-**discuss<http://openindiana.org/mailman/listinfo/openindiana-discuss>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the OpenIndiana-discuss
mailing list