Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X.X............X..........................................] duration:
23.819042792s
FAIL (2025-10-08 14:28:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 23.819081938s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X.X............X.........................................] duration:
7.631742411s
FAIL (2025-10-08 14:27:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.631776976s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X.X............X........................................] duration:
21.291622187s
FAIL (2025-10-08 14:26:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 21.291656632s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X.X............X.......................................] duration:
8.320497305s
FAIL (2025-10-08 14:25:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 8.320531431s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.X............X......................................] duration:
8.493416761s
FAIL (2025-10-08 14:25:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 8.493457768s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.X............X.....................................] duration:
6.590071318s
FAIL (2025-10-08 14:24:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.590101355s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.......X.X............X....................................] duration:
6.846532863s
FAIL (2025-10-08 14:23:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.846566126s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X.......X.X............X...................................] duration:
24.270885958s
FAIL (2025-10-08 14:22:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 24.270955921s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X.......X.X............X..................................] duration:
7.053560446s
FAIL (2025-10-08 14:21:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.053595062s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X.......X.X............X.................................] duration:
10.190997169s
FAIL (2025-10-08 14:20:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 10.191028718s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X...X.......X.X............X................................] duration:
30.002132312s
FAIL (2025-10-08 14:20:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 30.002171736s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X...X.......X.X............X...............................] duration:
19.55255458s
FAIL (2025-10-08 14:18:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 19.552598052s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X...X.......X.X............X..............................] duration:
6.346728136s
FAIL (2025-10-08 14:17:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.346759615s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X...X.......X.X............X.............................] duration:
16.485623715s
FAIL (2025-10-08 14:17:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 16.485659033s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X...X.......X.X............X............................] duration:
7.82310399s
FAIL (2025-10-08 14:16:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.823136042s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X...X.......X.X............X...........................] duration:
7.205188804s
FAIL (2025-10-08 14:14:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.205216065s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X...X.......X.X............X..........................] duration:
15.654983247s
FAIL (2025-10-08 14:14:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 15.655019807s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X...X.......X.X............X.........................] duration:
6.268877102s
FAIL (2025-10-08 14:13:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.268912479s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X...X.......X.X............X........................] duration:
6.419533079s
FAIL (2025-10-08 14:12:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.419566862s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X...X.......X.X............X.......................] duration:
12.985382978s
FAIL (2025-10-08 14:11:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 12.985416461s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X...X.......X.X............X......................] duration:
8.308611561s
FAIL (2025-10-08 14:10:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 8.308646466s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X...X.......X.X............X.....................] duration:
9.936948927s
FAIL (2025-10-08 14:10:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 9.93698227s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X...X.......X.X............X....................] duration:
26.834233219s
FAIL (2025-10-08 14:09:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 26.834261903s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X...X.......X.X............X...................] duration:
6.0143046s
FAIL (2025-10-08 14:08:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.014343425s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X...X.......X.X............X..................] duration:
11.473691748s
FAIL (2025-10-08 14:07:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 11.473739239s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X...X.......X.X............X.................] duration:
16.797980369s
FAIL (2025-10-08 14:06:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 16.798014493s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X...X.......X.X............X................] duration:
7.178192019s
FAIL (2025-10-08 14:05:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.178251912s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X...X.......X.X............X...............] duration:
18.902752185s
FAIL (2025-10-08 14:05:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 18.902790729s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X...X.......X.X............X..............] duration:
21.249520807s
FAIL (2025-10-08 14:04:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 21.24955907s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X...X.......X.X............X.............] duration:
12.786034944s
FAIL (2025-10-08 14:03:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 12.786071833s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X...................X...X.......X.X............X............] duration:
6.227858564s
FAIL (2025-10-08 14:02:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.227882039s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X...................X...X.......X.X............X...........] duration:
18.425835206s
FAIL (2025-10-08 14:02:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 18.425860262s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X...................X...X.......X.X............X..........] duration:
10.349338537s
FAIL (2025-10-08 14:01:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 10.349370479s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X...................X...X.......X.X............X.........] duration:
23.276210278s
FAIL (2025-10-08 14:01:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 23.276228583s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X...................X...X.......X.X............X........] duration:
28.461916002s
FAIL (2025-10-08 14:00:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 28.461942543s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X...................X...X.......X.X............X.......] duration:
6.603089598s
FAIL (2025-10-08 13:59:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.603135025s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X...................X...X.......X.X............X......] duration:
28.974460079s
FAIL (2025-10-08 13:58:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 28.974494465s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X...................X...X.......X.X............X.....] duration:
7.976545764s
FAIL (2025-10-08 13:57:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 7.976581562s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X...................X...X.......X.X............X....] duration:
28.914011222s
FAIL (2025-10-08 13:56:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 28.914040257s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X...................X...X.......X.X............X...] duration:
10.787305108s
FAIL (2025-10-08 13:55:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 10.787338141s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X...................X...X.......X.X............X..] duration:
18.907495815s
FAIL (2025-10-08 13:55:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 18.907526393s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........X...................X...X.......X.X............X.] duration:
6.412215059s
FAIL (2025-10-08 13:54:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 6.412239285s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X...................X...X.......X.X............X] duration:
30.002850935s
FAIL (2025-10-08 13:54:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA) context deadline exceeded
======LOGS======
Oct 8 13:53:40.767098557 (360h22m26.22919103s) [upload=317053]: Starting upload
Oct 8 13:53:40.767172227 (360h22m26.2292647s) [upload=317053]: Done starting upload
Oct 8 13:53:40.767275474 (360h22m26.229367967s) [upload=317053]: Uploading segments...
Oct 8 13:53:40.767400121 (360h22m26.229492604s) [upload=317053]: Got next segment. Inline: false
Oct 8 13:53:40.767454194 (360h22m26.229546677s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:53:40.767460376 (360h22m26.229552849s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:53:41.024435869 (360h22m26.486528342s) [seg_pos={0 0}, upload=317053]: Begin upload segment...
Oct 8 13:53:41.024489842 (360h22m26.486582315s) [seg_pos={0 0}, upload=317053]: Done begin upload segment.
Oct 8 13:53:41.024511663 (360h22m26.486604136s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024529037 (360h22m26.48662152s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.024589541 (360h22m26.486682014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025428710 (360h22m26.487521193s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.025858468 (360h22m26.487950941s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026283317 (360h22m26.48837579s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.026776696 (360h22m26.488869169s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027246752 (360h22m26.489339235s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.027497229 (360h22m26.489589712s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028463248 (360h22m26.490555721s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028590109 (360h22m26.490682582s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.028987917 (360h22m26.49108039s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029060926 (360h22m26.491153409s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.029882280 (360h22m26.491974773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030450983 (360h22m26.492543466s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:41.030873387 (360h22m26.49296586s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Uploading piece...
Oct 8 13:53:42.750796693 (360h22m28.212889186s) [upload=317053]: Next returned nil segment
Oct 8 13:53:42.750826069 (360h22m28.212918543s) [upload=317053]: Waiting for error group managing segments...
Oct 8 13:53:43.026194296 (360h22m28.488286789s) [noise=true , address=145.79.227.14:4838 , piece_id=KL25IUR3CNSZXVFDLILMZRU5KKR4NX3CQ3ZPU6KIHDOXALZS52LQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.059443709 (360h22m28.521536182s) [noise=true , address=145.79.227.11:4729 , piece_id=PH43DOMOWAJJ5EQNDKFXIGONTPXWEBTAA2VM2RJFC5H47CDVKHZA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.063750480 (360h22m28.525842963s) [noise=true , address=145.79.227.15:4874 , piece_id=SECCCWEFBVDUULJTKA4MZBDI4TC22VDYPBRGHS2HA3XZNL5YMZAQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.192755541 (360h22m28.654848014s) [noise=true , address=77.90.31.11:4555 , piece_id=4UQUBGR7HBB2PRKT2HSIZNM7KLI5FNPO7A2IL3QGNVPKVAER6IIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.212284449 (360h22m28.674376932s) [noise=true , address=145.79.227.19:4079 , piece_id=WX7DHREH7OGFWYVNESEDRM3WO72HCLUQ7373FITVZWM7WZYZTVXA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.255093195 (360h22m28.717185678s) [noise=true , address=45.91.116.7:4011 , piece_id=7ZPGNE4VB2YK2HBEVBK5ZUHLCDCG4N7PIGK3ZTPFM3FOCAEPZPIQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.278707772 (360h22m28.740800255s) [noise=true , address=77.90.31.19:4843 , piece_id=CQ3ZLFZ3YFY6DLZDCFNF4NPHOYTZX3A7DBOAS5JKF5ZX5W3E7WVA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.352664710 (360h22m28.814757193s) [noise=true , address=45.91.116.11:4118 , piece_id=RZUHDZP5UUAHNDQDMG2QBOSB7QG77PB4JIHVGYP7PIROPB6SIFQA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.403394220 (360h22m28.865486693s) [noise=true , address=45.91.116.8:4047 , piece_id=2KH6LZYJS2W3LZW6K4Z743HETETPSKBWBXKJ754ABVJMNQPAYA7Q, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.434180663 (360h22m28.896273136s) [noise=true , address=45.91.116.12:4227 , piece_id=OMI6KLXDRR7RAVFVAP43J4FL6G6FMQQZQLB6GWANDVCFEUQJLMKQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.448706073 (360h22m28.910798556s) [noise=true , address=45.91.116.5:4298 , piece_id=TZMFSJAUNZG4ATF5T27AVZMNRXF5FROGXFAQI6HHY33MBFKQQMAA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.560942483 (360h22m29.023034966s) [noise=true , address=45.91.116.13:4155 , piece_id=QLMILE6IZGX2QFC333NRR6DMJB6OMWOTZ4P6U6ASE3IQSU3OMHEA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691033922 (360h22m29.153126405s) [noise=true , address=45.91.116.9:4262 , piece_id=RJMDQRQ3DJ6G6FYOKN522SHIPNQYFNTQSLSHLOQXJEPU4BJ6XBTA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: <nil>
Oct 8 13:53:43.691064972 (360h22m29.153157445s) [seg_pos={0 0}, upload=317053]: Segment reached optimal threshold of 13 pieces.
Oct 8 13:53:43.691221370 (360h22m29.153313853s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37416->77.90.31.13:4626: use of closed network connection
Oct 8 13:53:43.691232240 (360h22m29.153324713s) [noise=true , address=77.90.31.13:4626 , piece_id=ID7AVKWC3JJZFJ5TSJHLGMBVPFLY6KHVF5VDFADWGRHC6OGEDMYQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691352810 (360h22m29.153445293s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691359563 (360h22m29.153452036s) [noise=true , address=45.91.116.11:4117 , piece_id=XNL32UFZT4Y6CWMB52HICEUXLBRB5OASNQCJQBPCUZZHTU7IF5TQ, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691419217 (360h22m29.15351169s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Oct 8 13:53:43.691424246 (360h22m29.153516719s) [noise=true , address=45.91.116.7:4009 , piece_id=RGNL5RSKTCH4OGAA7DROFCAHU5CTFDH3AV4KN6P6QAH6TDFFC2VA, seg_pos={0 0}, upload=317053]: Long tail cancellation due to optimal threshold
Oct 8 13:53:43.691437992 (360h22m29.153530465s) [seg_pos={0 0}, upload=317053]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 8 13:53:43.691500431 (360h22m29.153592914s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 8 13:53:43.691505020 (360h22m29.153597493s) [upload=317053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 8 13:53:44.008568521 (360h22m29.470661014s) [upload=317053]: Done uploading segments...
Oct 8 13:53:44.008594832 (360h22m29.470687325s) [upload=317053]: Upload finished. err: <nil>
Duration: 30.002875702s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............X...................X...X.......X.X............] duration:
13.377184334s
FAIL (2025-10-08 13:52:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 13.377236804s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............X...................X...X.......X.X...........] duration:
23.191252649s
FAIL (2025-10-08 13:52:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 23.191282056s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............X...................X...X.......X.X..........] duration:
6.390121907s
FAIL (2025-10-08 13:51:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.390164257s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X...................X...X.......X.X.........] duration:
26.366580461s
FAIL (2025-10-08 13:51:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 26.366621098s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X...................X...X.......X.X........] duration:
15.988138659s
FAIL (2025-10-08 13:49:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 15.988165901s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X.................X...................X...X.......X.X.......] duration:
29.125702236s
FAIL (2025-10-08 13:49:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 29.128135959s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X.................X...................X...X.......X.X......] duration:
6.93707269s
FAIL (2025-10-08 13:48:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 8 13:43:57.810063563 (360h12m43.272156046s) [upload=316904]: Starting upload
Oct 8 13:43:57.810137954 (360h12m43.272230427s) [upload=316904]: Done starting upload
Oct 8 13:43:57.810236702 (360h12m43.272329186s) [upload=316904]: Uploading segments...
Oct 8 13:43:57.810342684 (360h12m43.272435167s) [upload=316904]: Got next segment. Inline: false
Oct 8 13:43:57.810377731 (360h12m43.272470204s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 8 13:43:57.810383904 (360h12m43.272476387s) [upload=316904]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 8 13:43:58.063348919 (360h12m43.525441402s) [seg_pos={0 0}, upload=316904]: Begin upload segment...
Oct 8 13:43:58.063432057 (360h12m43.52552454s) [seg_pos={0 0}, upload=316904]: Done begin upload segment.
Oct 8 13:43:58.063454870 (360h12m43.525547353s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063485028 (360h12m43.525577511s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063542286 (360h12m43.525634769s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.063669789 (360h12m43.525762263s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064020457 (360h12m43.52611294s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.064261206 (360h12m43.526353699s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.065543117 (360h12m43.5276356s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066129293 (360h12m43.528221766s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.066674420 (360h12m43.528766904s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.071228953 (360h12m43.533321426s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.072519902 (360h12m43.534612385s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.074066978 (360h12m43.536159461s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.075496490 (360h12m43.537588963s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.077207959 (360h12m43.539300442s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078484940 (360h12m43.540577424s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:58.078901363 (360h12m43.540993847s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Uploading piece...
Oct 8 13:43:59.958821937 (360h12m45.42091442s) [upload=316904]: Next returned nil segment
Oct 8 13:43:59.958841474 (360h12m45.420933957s) [upload=316904]: Waiting for error group managing segments...
Oct 8 13:44:00.261248377 (360h12m45.72334087s) [noise=true , address=45.91.116.7:4011 , piece_id=NMQF7MFVVIR2EZYTPA7HJIM2JDEKQEE4FEJ2LLXNEU7MYBPPAVEA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.348925963 (360h12m45.811018446s) [noise=true , address=45.91.116.9:4263 , piece_id=2L2GE7VEU75NRTG5IZSK522TIJD63F4RGW63N7KKEBFRZ4FRYR6Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.385044931 (360h12m45.847137414s) [noise=true , address=45.91.116.8:4045 , piece_id=NMHOGB6U2AFEIU4S6MCXC47OIN2JV7YFUYGZXDXW23T4KDKYACQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.392667596 (360h12m45.854760069s) [noise=true , address=45.91.116.8:4048 , piece_id=DVQQ6JZAKWDJLT7UUI44ZPNW2DLDV2U74QTVJ5BSDNQ2LCB6XUKA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.460578662 (360h12m45.922671135s) [noise=true , address=45.91.116.6:4975 , piece_id=R57Y7ZWZINGHLO3LU6EFTU75OKM7U55OZHNPX3P34G22HE2YAQEQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.534763896 (360h12m45.996856379s) [noise=true , address=77.90.31.15:4699 , piece_id=QWUEOYCT5A5HRF7ESXIALESERCVQ7YYEOUOKXDTFBX63V3H355YQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:00.620447514 (360h12m46.082539997s) [noise=true , address=45.91.116.13:4154 , piece_id=FCH4E4CQXL5GBRQLYLYFMUQR6RBMXIL6GODUQAYIG6Q4IEOAP32Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.006762044 (360h12m46.468854527s) [noise=true , address=77.90.31.21:4877 , piece_id=L77FIU6B6LDSNUDNKL4RFT3HXRBTR3GRGLEASYLNA4EHO6EJPFGA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.034533997 (360h12m46.49662647s) [noise=true , address=45.91.116.14:4190 , piece_id=AURDR45DCGBPGVVQT62K5ONK5RTIPKILD4QQFIYZHEIREO2LEQMQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:01.191701205 (360h12m46.653793698s) [noise=true , address=77.90.31.15:4698 , piece_id=CZRL5SCLI4BYBL2EELMIZ2AA7COJKTKP2AIMNJ2GHEC7MIIZMEQA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:16.644858234 (360h13m2.106950717s) [noise=true , address=145.79.227.15:4876 , piece_id=PXGEHOOAKBWL3F7KMVM2U6HHGP2LGWMWJJGMRZJSEYSYWLNUQVQQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:26.364996329 (360h13m11.827088812s) [noise=true , address=145.79.227.14:4840 , piece_id=3LASRSQFMGSQLZBQ2FTAC32AVVTAWJIMUXHAS5GJLB4EDBTORWUQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: <nil>
Oct 8 13:44:27.082525848 (360h13m12.544618321s) [noise=true , address=145.79.227.17:4971 , piece_id=EJGPI2UHIWIUIIDQ4U3C7LAK2HXAUHONFBO5KKUQGVUPZTZ7GFXQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34576->145.79.227.17:4971: use of closed network connection
Oct 8 13:44:27.082668189 (360h13m12.544760662s) [noise=true , address=145.79.227.4:4185 , piece_id=O55EYVXFJQHZHB2HVPP2CUG27PWVHBYKYD25AMGNZYESX73GEDVQ, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:145.79.227.4:4185): protocol: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56886->145.79.227.4:4185: use of closed network connection
Oct 8 13:44:27.082730177 (360h13m12.54482266s) [noise=true , address=145.79.227.11:4730 , piece_id=PZ5B7E4FYKQ53YXJXQ55B6C6EHEOM6PCZOPEOT2KHY4EXTZRJRCA, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.11:4730: use of closed network connection
Oct 8 13:44:27.082781424 (360h13m12.544873897s) [noise=true , address=145.79.227.13:4804 , piece_id=QBMIVIXYB5J35BDYTUBJ3FLFCUOECTRMLKBDE4YTYA5K3PIXAY7Q, seg_pos={0 0}, upload=316904]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56040->145.79.227.13:4804: use of closed network connection
Oct 8 13:44:27.082802865 (360h13m12.544895338s) [seg_pos={0 0}, upload=316904]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 8 13:44:27.341447707 (360h13m12.80354021s) [upload=316904]: Done uploading segments...
Oct 8 13:44:27.341473306 (360h13m12.803565789s) [upload=316904]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.937133376s
core
uplink
us-select-1
|