Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....X] duration:
30.002587084s trace:
https://jaeger.dev.storj.io/trace/61cbddb630affadb
FAIL (2025-12-31 03:45:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 3KKKXM4442MPPA7HDD745GD3QIRFPMI4WH5L5SLKTX7ATXBOP3BA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ADJRF776L3NY66KV3EZN2ESKTUZ5CRPRAXG4JT3HURLVOJ6IOMUQ) context deadline exceeded
======LOGS======
ec 31 03:45:29.413023308 (612h16m29.284106353s) [upload=536564]: Starting upload
Dec 31 03:45:29.413145431 (612h16m29.284228466s) [upload=536564]: Done starting upload
Dec 31 03:45:29.413241214 (612h16m29.284324249s) [upload=536564]: Uploading segments...
Dec 31 03:45:29.413522640 (612h16m29.284605675s) [upload=536564]: Got next segment. Inline: false
Dec 31 03:45:29.413574518 (612h16m29.284657563s) [upload=536564]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:45:29.413580910 (612h16m29.284663975s) [upload=536564]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:45:29.659374149 (612h16m29.530457194s) [seg_pos={0 0}, upload=536564]: Begin upload segment...
Dec 31 03:45:29.659433001 (612h16m29.530516047s) [seg_pos={0 0}, upload=536564]: Done begin upload segment.
Dec 31 03:45:29.659461916 (612h16m29.530544961s) [noise=true , address=145.79.227.10:4402 , piece_id=4IVCVDY2QGUKWYDSBMTL7OXSMMMVOJMNUGKHWV25CIKDKUFYJXZQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.661997654 (612h16m29.533080689s) [noise=true , address=45.91.116.4:4940 , piece_id=LDPMHXASLHELWC3DHQR7HCLF4K2YQQ4YGIFJ7I343OLN5U2FURGQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.662766178 (612h16m29.533849223s) [noise=true , address=145.79.227.20:4437 , piece_id=5SYV4EC2BV2IR3LH6A6H45IPEDVP7N35PLRTQZC3FDKRMRYN367A, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.663160869 (612h16m29.534243904s) [noise=true , address=45.91.116.8:4045 , piece_id=N5FU4PIG5BUJNBPFGVOHNLGT2XIGILXT5SOSG5DG274B6LAJ2XMQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.664174680 (612h16m29.535257715s) [noise=true , address=45.91.116.11:4120 , piece_id=266BHQ5OEPJAOBJRHVRCAS6U6X7SMQYFV4T3V4HSCDSSNOC3YCLQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.665188370 (612h16m29.536271405s) [noise=true , address=77.90.31.10:4520 , piece_id=P7VVJLCQCSCLYG2QM57BZ7T7LHEWJFKOSZIPWBD4Y4FLJ2RYJP6A, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.665440530 (612h16m29.536523565s) [noise=true , address=145.79.227.37:4565 , piece_id=OFZNXMFFOBSA5O5YQ46Y3ZWYOSWD7KPLRO5L2V2YTYYWQ7MZEWZA, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.667124917 (612h16m29.538207962s) [noise=true , address=145.79.227.29:4941 , piece_id=RPMB423NOHGVQZEALR47O3XO6TSTWFR6QQFDUF65OPCBE45YFV6Q, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.667240838 (612h16m29.538323873s) [noise=true , address=145.79.227.9:4365 , piece_id=3QGQY5TZ33GKWDZWU6CO6J7L6LUAE7NWRBG6JOEHSMO3BE7AB2RA, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.669133813 (612h16m29.540216858s) [noise=true , address=145.79.227.8:4329 , piece_id=QQU25DDFUWIUIAPQWW2W6GOZNR5KIIIASYZE5GLD5DQJBJI67MPA, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.669159080 (612h16m29.540242115s) [noise=true , address=45.91.116.9:4263 , piece_id=ADJRF776L3NY66KV3EZN2ESKTUZ5CRPRAXG4JT3HURLVOJ6IOMUQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.671289336 (612h16m29.542372381s) [noise=true , address=145.79.227.31:4798 , piece_id=CIXVQA5FTR4WUHMPEHZYHHLEB3RTCT7JVB5I5TVHFYJ42NSZG3QQ, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.672606715 (612h16m29.54368977s) [noise=true , address=145.79.227.52:4447 , piece_id=6E7GOX4ZPNWBEIZTXLISKNWJQ4WMH4Y6MNPYQDVRPFK4AE2J3DDA, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.675030100 (612h16m29.546113145s) [noise=true , address=77.90.31.14:4662 , piece_id=WYJGQBR5DSFX66MW5444DJKMSLLPIRVVGGMOI6PABS22U5GH4Z3Q, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.675169586 (612h16m29.546252621s) [noise=true , address=145.79.227.11:4729 , piece_id=KP5ATUNEVFXYFH5OOWRMUVDM2BSND6MUXPRCMVP5KDHK2NFW266A, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:29.688724125 (612h16m29.55980717s) [noise=true , address=77.90.31.18:4805 , piece_id=3KKKXM4442MPPA7HDD745GD3QIRFPMI4WH5L5SLKTX7ATXBOP3BA, seg_pos={0 0}, upload=536564]: Uploading piece...
Dec 31 03:45:31.266435468 (612h16m31.137518513s) [upload=536564]: Next returned nil segment
Dec 31 03:45:31.266454264 (612h16m31.137537309s) [upload=536564]: Waiting for error group managing segments...
Dec 31 03:45:31.649397755 (612h16m31.5204808s) [noise=true , address=145.79.227.20:4437 , piece_id=5SYV4EC2BV2IR3LH6A6H45IPEDVP7N35PLRTQZC3FDKRMRYN367A, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.675138949 (612h16m31.546221984s) [noise=true , address=145.79.227.37:4565 , piece_id=OFZNXMFFOBSA5O5YQ46Y3ZWYOSWD7KPLRO5L2V2YTYYWQ7MZEWZA, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.677000233 (612h16m31.548083268s) [noise=true , address=145.79.227.9:4365 , piece_id=3QGQY5TZ33GKWDZWU6CO6J7L6LUAE7NWRBG6JOEHSMO3BE7AB2RA, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.685144781 (612h16m31.556227817s) [noise=true , address=145.79.227.10:4402 , piece_id=4IVCVDY2QGUKWYDSBMTL7OXSMMMVOJMNUGKHWV25CIKDKUFYJXZQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.685556915 (612h16m31.55663995s) [noise=true , address=145.79.227.31:4798 , piece_id=CIXVQA5FTR4WUHMPEHZYHHLEB3RTCT7JVB5I5TVHFYJ42NSZG3QQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.695812092 (612h16m31.566895137s) [noise=true , address=145.79.227.52:4447 , piece_id=6E7GOX4ZPNWBEIZTXLISKNWJQ4WMH4Y6MNPYQDVRPFK4AE2J3DDA, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.700152067 (612h16m31.571235102s) [noise=true , address=145.79.227.8:4329 , piece_id=QQU25DDFUWIUIAPQWW2W6GOZNR5KIIIASYZE5GLD5DQJBJI67MPA, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.702245152 (612h16m31.573328207s) [noise=true , address=145.79.227.11:4729 , piece_id=KP5ATUNEVFXYFH5OOWRMUVDM2BSND6MUXPRCMVP5KDHK2NFW266A, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.957792274 (612h16m31.828875309s) [noise=true , address=77.90.31.14:4662 , piece_id=WYJGQBR5DSFX66MW5444DJKMSLLPIRVVGGMOI6PABS22U5GH4Z3Q, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.968201835 (612h16m31.83928487s) [noise=true , address=77.90.31.10:4520 , piece_id=P7VVJLCQCSCLYG2QM57BZ7T7LHEWJFKOSZIPWBD4Y4FLJ2RYJP6A, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:31.979556195 (612h16m31.85063927s) [noise=true , address=77.90.31.18:4805 , piece_id=3KKKXM4442MPPA7HDD745GD3QIRFPMI4WH5L5SLKTX7ATXBOP3BA, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:32.104032143 (612h16m31.975115188s) [noise=true , address=45.91.116.4:4940 , piece_id=LDPMHXASLHELWC3DHQR7HCLF4K2YQQ4YGIFJ7I343OLN5U2FURGQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:32.322636603 (612h16m32.193719648s) [noise=true , address=45.91.116.9:4263 , piece_id=ADJRF776L3NY66KV3EZN2ESKTUZ5CRPRAXG4JT3HURLVOJ6IOMUQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: <nil>
Dec 31 03:45:32.322660979 (612h16m32.193744014s) [seg_pos={0 0}, upload=536564]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:45:32.322936483 (612h16m32.194019518s) [noise=true , address=45.91.116.11:4120 , piece_id=266BHQ5OEPJAOBJRHVRCAS6U6X7SMQYFV4T3V4HSCDSSNOC3YCLQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:48624->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:48624->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48624->45.91.116.11:4120: use of closed network connection
Dec 31 03:45:32.322944879 (612h16m32.194027934s) [noise=true , address=45.91.116.11:4120 , piece_id=266BHQ5OEPJAOBJRHVRCAS6U6X7SMQYFV4T3V4HSCDSSNOC3YCLQ, seg_pos={0 0}, upload=536564]: Long tail cancellation due to optimal threshold
Dec 31 03:45:32.323182722 (612h16m32.194265767s) [noise=true , address=145.79.227.29:4941 , piece_id=RPMB423NOHGVQZEALR47O3XO6TSTWFR6QQFDUF65OPCBE45YFV6Q, seg_pos={0 0}, upload=536564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 31 03:45:32.323198553 (612h16m32.194281598s) [noise=true , address=45.91.116.8:4045 , piece_id=N5FU4PIG5BUJNBPFGVOHNLGT2XIGILXT5SOSG5DG274B6LAJ2XMQ, seg_pos={0 0}, upload=536564]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:45:32.323211577 (612h16m32.194294622s) [noise=true , address=45.91.116.8:4045 , piece_id=N5FU4PIG5BUJNBPFGVOHNLGT2XIGILXT5SOSG5DG274B6LAJ2XMQ, seg_pos={0 0}, upload=536564]: Long tail cancellation due to optimal threshold
Dec 31 03:45:32.323222287 (612h16m32.194305322s) [noise=true , address=145.79.227.29:4941 , piece_id=RPMB423NOHGVQZEALR47O3XO6TSTWFR6QQFDUF65OPCBE45YFV6Q, seg_pos={0 0}, upload=536564]: Long tail cancellation due to optimal threshold
Dec 31 03:45:32.323392071 (612h16m32.194475106s) [seg_pos={0 0}, upload=536564]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:45:32.323477815 (612h16m32.19456085s) [upload=536564]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:45:32.323483415 (612h16m32.19456645s) [upload=536564]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:45:32.609597411 (612h16m32.480680456s) [upload=536564]: Done uploading segments...
Dec 31 03:45:32.609621647 (612h16m32.480704692s) [upload=536564]: Upload finished. err: <nil>
Duration: 30.002611291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [......X.......X.X...X...X...X.X.X...X......X..XX.....X....X.] duration:
3.369201439s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:45:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.36922868s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X....X.X..X...................] duration:
6.765287206s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:45:17 UTC)
orbiter warning: context deadline exceeded
Duration: 6.765315438s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
42.279239744s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:45:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 42.279270121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....] duration:
6.100255836s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:44:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 6.100281685s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.......X.......X.X...X...X...X.X.X...X......X..XX.....X....X] duration:
5.001315321s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:44:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001337873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X....X.X..X..................] duration:
12.063395687s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:44:25 UTC)
orbiter warning: context deadline exceeded
Duration: 12.063413643s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X....] duration:
5.951535807s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:44:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 5.951567829s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
46.609062954s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:44:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 46.609100987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X....X.X..X.................] duration:
10.889030774s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:44:01 UTC)
orbiter warning: context deadline exceeded
Duration: 10.889057364s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [........X.......X.X...X...X...X.X.X...X......X..XX.....X....] duration:
3.509980525s trace:
https://jaeger.dev.storj.io/trace/345891df18f57a78
FAIL (2025-12-31 03:43:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.510008059s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X...] duration:
5.695954976s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:43:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 5.695980294s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X....X.X..X................] duration:
6.41133389s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:43:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.411354619s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.........X.......X.X...X...X...X.X.X...X......X..XX.....X...] duration:
3.531588672s trace:
https://jaeger.dev.storj.io/trace/345891df18f57a78
FAIL (2025-12-31 03:42:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.531620043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
15.388394648s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:42:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 15.388412491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
47.181828415s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:42:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 47.181865176s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................X....X.X..X...............] duration:
6.647214071s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:42:16 UTC)
orbiter warning: context deadline exceeded
Duration: 6.647242907s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X..] duration:
5.895030281s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:42:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 5.895086677s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X....X.X..X..............] duration:
13.495407943s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:42:08 UTC)
orbiter warning: context deadline exceeded
Duration: 13.495434884s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [..........X.......X.X...X...X...X.X.X...X......X..XX.....X..] duration:
3.444443398s trace:
https://jaeger.dev.storj.io/trace/345891df18f57a78
FAIL (2025-12-31 03:42:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.444474336s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.] duration:
5.774715377s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:41:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 5.774736969s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X] duration:
30.005009075s trace:
https://jaeger.dev.storj.io/trace/7637b10427307708
FAIL (2025-12-31 03:41:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA) context deadline exceeded
======LOGS======
ec 31 03:41:15.484309097 (612h12m15.355392132s) [upload=536500]: Starting upload
Dec 31 03:41:15.484364271 (612h12m15.355447306s) [upload=536500]: Done starting upload
Dec 31 03:41:15.484540858 (612h12m15.355623893s) [upload=536500]: Uploading segments...
Dec 31 03:41:15.484587046 (612h12m15.355670081s) [upload=536500]: Got next segment. Inline: false
Dec 31 03:41:15.484611012 (612h12m15.355694047s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:41:15.484616152 (612h12m15.355699197s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:41:15.733200927 (612h12m15.604283962s) [seg_pos={0 0}, upload=536500]: Begin upload segment...
Dec 31 03:41:15.733248048 (612h12m15.604331083s) [seg_pos={0 0}, upload=536500]: Done begin upload segment.
Dec 31 03:41:15.733269177 (612h12m15.604352212s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733322840 (612h12m15.604405885s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733659600 (612h12m15.604742635s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.733950856 (612h12m15.605033891s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734273490 (612h12m15.605356525s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734694792 (612h12m15.605777837s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.734959526 (612h12m15.606042571s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735303902 (612h12m15.606386937s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.735831066 (612h12m15.606914101s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.736290831 (612h12m15.607373866s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.737680157 (612h12m15.608763192s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.739329888 (612h12m15.610412933s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.741235126 (612h12m15.612318161s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.747669527 (612h12m15.618752562s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.749554697 (612h12m15.620637742s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:15.751876889 (612h12m15.622959924s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Uploading piece...
Dec 31 03:41:17.434303835 (612h12m17.30538687s) [upload=536500]: Next returned nil segment
Dec 31 03:41:17.434321808 (612h12m17.305404854s) [upload=536500]: Waiting for error group managing segments...
Dec 31 03:41:17.718293745 (612h12m17.589376781s) [noise=true , address=145.79.227.18:4007 , piece_id=MAOMFLUTL4J4OZ22INPYYFYHDL3XCWBXT4RMCH4G4FR6X4VPDDEQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.720635554 (612h12m17.591718599s) [noise=true , address=145.79.227.41:4087 , piece_id=YLYUDBVEK2J7PW4LG6KKLE75WJZYZC7ZSQFB4NZMZSMF7F6OYAPQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735476584 (612h12m17.606559639s) [noise=true , address=145.79.227.10:4403 , piece_id=IWDTAQ6ZBYQWUID64TKOWRHU7FDHU3GNUZN7XMIVNIGZ7MQG5CRQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.735829606 (612h12m17.606912651s) [noise=true , address=145.79.227.12:4767 , piece_id=53FRNX3OBQRS76CUZWQSEA6UVVP6BZMOFLKOEXAO73SEBAMJAI6A, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.737239000 (612h12m17.608322035s) [noise=true , address=145.79.227.45:4229 , piece_id=OTOP5TSQPT6WNKKONRVXG3FL5MR62IFGILD4GDQDYI7NDP63QUVA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.752398174 (612h12m17.62348121s) [noise=true , address=145.79.227.14:4839 , piece_id=SIEEEGPAWROGIIHPIFCVKKH5WSVNMC5GJHLVTPJKSZJJZ6YUFFLQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.753022333 (612h12m17.624105378s) [noise=true , address=145.79.227.15:4874 , piece_id=7BCDZ6WYSZ3QPYUDBBHYYWTEVZ2HZC7SEVX4QMQEC6Z6SAJASTHQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.830466175 (612h12m17.70154922s) [noise=true , address=145.79.227.31:4798 , piece_id=UW4VFKF3C5EZEGDVH74DMCUXZL4JUBD2PS2XBKTPYCHW6XQMDUCA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.898095099 (612h12m17.769178134s) [noise=true , address=77.90.31.16:4735 , piece_id=ACOW2NHRTVT626MIHUVEMJA4DTVKVYKVZAM52JYZS4BGDMCNO5MA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.937786687 (612h12m17.808869732s) [noise=true , address=77.90.31.13:4629 , piece_id=32EEQ5PU7EGENYO2WDH6BHDQUP54PIA52O5XWX7FZVVW5CFKRDKA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:17.953959933 (612h12m17.825042978s) [noise=true , address=77.90.31.18:4805 , piece_id=ZMMMENQAESN5IPCM6YI4SQVWOFH4LJFMGZRPRL4JZXWFRHOAQ3GA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.032229317 (612h12m17.903312362s) [noise=true , address=77.90.31.14:4661 , piece_id=VWWCKD5GWCMFFW7ULL4V4IHFDVA4U7RSCJUOMFREKUSI5N6THDGA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626755876 (612h12m18.497838921s) [noise=true , address=77.90.31.11:4555 , piece_id=RFHKLLKRFOKNAW6WUUEQQHA6ARLNRRV2Z3UF2JT5KAC3RZXEK2AQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: <nil>
Dec 31 03:41:18.626775493 (612h12m18.497858528s) [seg_pos={0 0}, upload=536500]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:41:18.626993008 (612h12m18.498076053s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.626999951 (612h12m18.498082986s) [noise=true , address=77.90.31.12:4590 , piece_id=QJ22OJNL6PKNJ2DGO7WOLNCM2EOPK4G3NEXHV6EUHH7FZLBLMXIQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627152721 (612h12m18.498235756s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->45.91.116.7:4010: use of closed network connection
Dec 31 03:41:18.627175405 (612h12m18.49825849s) [noise=true , address=45.91.116.7:4010 , piece_id=DGOS67QNOY65LVWQDJNBDTM2YRNDOBX3XTUIYU2V4KDFGJINNOOA, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627212185 (612h12m18.49829522s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF
Dec 31 03:41:18.627220460 (612h12m18.498303505s) [noise=true , address=45.91.116.10:4081 , piece_id=KKGHEQOYN7QUPMEJDII2KWHHD4KSB5IPHKVDF74ECQH5ZIOTDZFQ, seg_pos={0 0}, upload=536500]: Long tail cancellation due to optimal threshold
Dec 31 03:41:18.627239076 (612h12m18.498322111s) [seg_pos={0 0}, upload=536500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:41:18.627321803 (612h12m18.498404839s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:41:18.627327013 (612h12m18.498410048s) [upload=536500]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:41:18.925242130 (612h12m18.796325175s) [upload=536500]: Done uploading segments...
Dec 31 03:41:18.925263581 (612h12m18.796346616s) [upload=536500]: Upload finished. err: <nil>
Duration: 30.005037488s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [X..........X.......X.X...X...X...X.X.X...X......X..XX.....X.] duration:
5.001133453s trace:
https://jaeger.dev.storj.io/trace/345891df18f57a78
FAIL (2025-12-31 03:41:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001161126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................X....X.X..X.............] duration:
15.429212147s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:41:23 UTC)
orbiter warning: context deadline exceeded
Duration: 15.429236834s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
50.241155238s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:41:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 50.241172911s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.X..........X.......X.X...X...X...X.X.X...X......X..XX.....X] duration:
5.001861759s trace:
https://jaeger.dev.storj.io/trace/345891df18f57a78
FAIL (2025-12-31 03:41:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001903499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.] duration:
6.918868884s trace:
https://jaeger.dev.storj.io/trace/10a70ece1cde8e9c
FAIL (2025-12-31 03:40:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ) context deadline exceeded
======LOGS======
Dec 31 03:40:06.323105441 (612h11m6.194188486s) [upload=536485]: Starting upload
Dec 31 03:40:06.323290403 (612h11m6.194373439s) [upload=536485]: Done starting upload
Dec 31 03:40:06.323344105 (612h11m6.19442715s) [upload=536485]: Uploading segments...
Dec 31 03:40:06.323545890 (612h11m6.194628935s) [upload=536485]: Got next segment. Inline: false
Dec 31 03:40:06.323593330 (612h11m6.194676385s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:40:06.323602598 (612h11m6.194685643s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:40:06.575953303 (612h11m6.447036358s) [seg_pos={0 0}, upload=536485]: Begin upload segment...
Dec 31 03:40:06.576023407 (612h11m6.447106452s) [seg_pos={0 0}, upload=536485]: Done begin upload segment.
Dec 31 03:40:06.576066860 (612h11m6.447149905s) [noise=true , address=145.79.227.38:4978 , piece_id=4SL7EQLMSAI4GNHJTEGXFT3TJTNBIIWIBGJK2CLUX2SE5PB3442A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576080165 (612h11m6.44716321s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576144237 (612h11m6.447227272s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576829232 (612h11m6.447912277s) [noise=true , address=145.79.227.37:4565 , piece_id=RICSFV3RXT5TKVY2RU7YU7ODVH5YMD653IA6HOMJNY535WILWHZA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576977013 (612h11m6.448060058s) [noise=true , address=145.79.227.47:4265 , piece_id=AEBKVR5SY6FS6Y7OYZHPJ4TXYTHMAHRTAG6DWMCI25OITOOCI37A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.577302042 (612h11m6.448385087s) [noise=true , address=145.79.227.41:4085 , piece_id=AEMT4DDHEGZWUUKYPW2VH74L3RHCV7X7KNYFPEEU7HHOGU63V5YQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.577607443 (612h11m6.448690488s) [noise=true , address=145.79.227.32:4836 , piece_id=Y2AWRUHQENFG5QPYCHNO5SUZADPKQPAKVKVBOJPQSQ6TUSIF5KMQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578093088 (612h11m6.449176133s) [noise=true , address=145.79.227.9:4365 , piece_id=FHFKJ5BTRAQBJTARN67A6XBNEM7N7NHC27UMAUV6IULSKA3BGY2A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578109038 (612h11m6.449192083s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578164664 (612h11m6.449247699s) [noise=true , address=145.79.227.28:4726 , piece_id=G7BHMGKGOUMJIWGFLIDPOFVMDDQEFREOBQ4VCT2MPJO5AGADRZWA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578713830 (612h11m6.449796875s) [noise=true , address=45.91.116.11:4118 , piece_id=PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579164989 (612h11m6.450248045s) [noise=true , address=145.79.227.2:4113 , piece_id=CIZ6BPUTRWYEKHO64DG3VUCA7E57VR67DSKMBRT2Z5PHHBZDEMRQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579538730 (612h11m6.450621775s) [noise=true , address=145.79.227.7:4295 , piece_id=5VZO7W62ISF4AXICMG67HIVXKUQP7UGS2KTFBMB3DUKHY2CM2ONA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579918103 (612h11m6.451001148s) [noise=true , address=145.79.227.45:4229 , piece_id=CAWGMIKUIFWQHDFGTGJZWBNEOIIU6NOKNWI5FYDOHIAEI3ZTOUWQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.580314197 (612h11m6.451397232s) [noise=true , address=145.79.227.14:4838 , piece_id=C2KGOTGETVBAROCJ7E3I4HHULBW4H72D3F7BEKPVUBPKJSBWHXAQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.580822946 (612h11m6.451905991s) [noise=true , address=77.90.31.14:4662 , piece_id=YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:08.271585393 (612h11m8.142668439s) [upload=536485]: Next returned nil segment
Dec 31 03:40:08.271601914 (612h11m8.14268496s) [upload=536485]: Waiting for error group managing segments...
Dec 31 03:40:08.558587229 (612h11m8.429670264s) [noise=true , address=145.79.227.2:4113 , piece_id=CIZ6BPUTRWYEKHO64DG3VUCA7E57VR67DSKMBRT2Z5PHHBZDEMRQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.559193263 (612h11m8.430276309s) [noise=true , address=145.79.227.38:4978 , piece_id=4SL7EQLMSAI4GNHJTEGXFT3TJTNBIIWIBGJK2CLUX2SE5PB3442A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.563806648 (612h11m8.434889693s) [noise=true , address=145.79.227.45:4229 , piece_id=CAWGMIKUIFWQHDFGTGJZWBNEOIIU6NOKNWI5FYDOHIAEI3ZTOUWQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.565585525 (612h11m8.43666857s) [noise=true , address=145.79.227.32:4836 , piece_id=Y2AWRUHQENFG5QPYCHNO5SUZADPKQPAKVKVBOJPQSQ6TUSIF5KMQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.565709672 (612h11m8.436792707s) [noise=true , address=145.79.227.28:4726 , piece_id=G7BHMGKGOUMJIWGFLIDPOFVMDDQEFREOBQ4VCT2MPJO5AGADRZWA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.574173587 (612h11m8.445256632s) [noise=true , address=145.79.227.37:4565 , piece_id=RICSFV3RXT5TKVY2RU7YU7ODVH5YMD653IA6HOMJNY535WILWHZA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.581151284 (612h11m8.452234319s) [noise=true , address=145.79.227.9:4365 , piece_id=FHFKJ5BTRAQBJTARN67A6XBNEM7N7NHC27UMAUV6IULSKA3BGY2A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.588227949 (612h11m8.459310995s) [noise=true , address=145.79.227.14:4838 , piece_id=C2KGOTGETVBAROCJ7E3I4HHULBW4H72D3F7BEKPVUBPKJSBWHXAQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.636319211 (612h11m8.507402256s) [noise=true , address=145.79.227.7:4295 , piece_id=5VZO7W62ISF4AXICMG67HIVXKUQP7UGS2KTFBMB3DUKHY2CM2ONA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.640095251 (612h11m8.511178296s) [noise=true , address=145.79.227.47:4265 , piece_id=AEBKVR5SY6FS6Y7OYZHPJ4TXYTHMAHRTAG6DWMCI25OITOOCI37A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.645671579 (612h11m8.516754654s) [noise=true , address=145.79.227.41:4085 , piece_id=AEMT4DDHEGZWUUKYPW2VH74L3RHCV7X7KNYFPEEU7HHOGU63V5YQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.828391169 (612h11m8.699474204s) [noise=true , address=77.90.31.14:4662 , piece_id=YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.906771956 (612h11m8.777854991s) [noise=true , address=45.91.116.11:4118 , piece_id=PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.906790982 (612h11m8.777874017s) [seg_pos={0 0}, upload=536485]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:40:08.906993227 (612h11m8.778076272s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907000601 (612h11m8.778083636s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907018775 (612h11m8.778101821s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907031459 (612h11m8.778114504s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907378109 (612h11m8.778461144s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907385304 (612h11m8.778468339s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907533285 (612h11m8.77861632s) [seg_pos={0 0}, upload=536485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:40:08.907608989 (612h11m8.778692024s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:40:08.907613688 (612h11m8.778696723s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:40:09.204553217 (612h11m9.075636262s) [upload=536485]: Done uploading segments...
Dec 31 03:40:09.204578726 (612h11m9.075661761s) [upload=536485]: Upload finished. err: <nil>
Duration: 6.918895083s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................X....X.X..X............] duration:
21.812934577s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:40:41 UTC)
orbiter warning: context deadline exceeded
Duration: 21.812964253s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXXX] duration:
30.005501996s trace:
https://jaeger.dev.storj.io/trace/10a70ece1cde8e9c
FAIL (2025-12-31 03:40:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ) context deadline exceeded
======LOGS======
Dec 31 03:40:06.323105441 (612h11m6.194188486s) [upload=536485]: Starting upload
Dec 31 03:40:06.323290403 (612h11m6.194373439s) [upload=536485]: Done starting upload
Dec 31 03:40:06.323344105 (612h11m6.19442715s) [upload=536485]: Uploading segments...
Dec 31 03:40:06.323545890 (612h11m6.194628935s) [upload=536485]: Got next segment. Inline: false
Dec 31 03:40:06.323593330 (612h11m6.194676385s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:40:06.323602598 (612h11m6.194685643s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:40:06.575953303 (612h11m6.447036358s) [seg_pos={0 0}, upload=536485]: Begin upload segment...
Dec 31 03:40:06.576023407 (612h11m6.447106452s) [seg_pos={0 0}, upload=536485]: Done begin upload segment.
Dec 31 03:40:06.576066860 (612h11m6.447149905s) [noise=true , address=145.79.227.38:4978 , piece_id=4SL7EQLMSAI4GNHJTEGXFT3TJTNBIIWIBGJK2CLUX2SE5PB3442A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576080165 (612h11m6.44716321s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576144237 (612h11m6.447227272s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576829232 (612h11m6.447912277s) [noise=true , address=145.79.227.37:4565 , piece_id=RICSFV3RXT5TKVY2RU7YU7ODVH5YMD653IA6HOMJNY535WILWHZA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.576977013 (612h11m6.448060058s) [noise=true , address=145.79.227.47:4265 , piece_id=AEBKVR5SY6FS6Y7OYZHPJ4TXYTHMAHRTAG6DWMCI25OITOOCI37A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.577302042 (612h11m6.448385087s) [noise=true , address=145.79.227.41:4085 , piece_id=AEMT4DDHEGZWUUKYPW2VH74L3RHCV7X7KNYFPEEU7HHOGU63V5YQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.577607443 (612h11m6.448690488s) [noise=true , address=145.79.227.32:4836 , piece_id=Y2AWRUHQENFG5QPYCHNO5SUZADPKQPAKVKVBOJPQSQ6TUSIF5KMQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578093088 (612h11m6.449176133s) [noise=true , address=145.79.227.9:4365 , piece_id=FHFKJ5BTRAQBJTARN67A6XBNEM7N7NHC27UMAUV6IULSKA3BGY2A, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578109038 (612h11m6.449192083s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578164664 (612h11m6.449247699s) [noise=true , address=145.79.227.28:4726 , piece_id=G7BHMGKGOUMJIWGFLIDPOFVMDDQEFREOBQ4VCT2MPJO5AGADRZWA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.578713830 (612h11m6.449796875s) [noise=true , address=45.91.116.11:4118 , piece_id=PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579164989 (612h11m6.450248045s) [noise=true , address=145.79.227.2:4113 , piece_id=CIZ6BPUTRWYEKHO64DG3VUCA7E57VR67DSKMBRT2Z5PHHBZDEMRQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579538730 (612h11m6.450621775s) [noise=true , address=145.79.227.7:4295 , piece_id=5VZO7W62ISF4AXICMG67HIVXKUQP7UGS2KTFBMB3DUKHY2CM2ONA, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.579918103 (612h11m6.451001148s) [noise=true , address=145.79.227.45:4229 , piece_id=CAWGMIKUIFWQHDFGTGJZWBNEOIIU6NOKNWI5FYDOHIAEI3ZTOUWQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.580314197 (612h11m6.451397232s) [noise=true , address=145.79.227.14:4838 , piece_id=C2KGOTGETVBAROCJ7E3I4HHULBW4H72D3F7BEKPVUBPKJSBWHXAQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:06.580822946 (612h11m6.451905991s) [noise=true , address=77.90.31.14:4662 , piece_id=YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ, seg_pos={0 0}, upload=536485]: Uploading piece...
Dec 31 03:40:08.271585393 (612h11m8.142668439s) [upload=536485]: Next returned nil segment
Dec 31 03:40:08.271601914 (612h11m8.14268496s) [upload=536485]: Waiting for error group managing segments...
Dec 31 03:40:08.558587229 (612h11m8.429670264s) [noise=true , address=145.79.227.2:4113 , piece_id=CIZ6BPUTRWYEKHO64DG3VUCA7E57VR67DSKMBRT2Z5PHHBZDEMRQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.559193263 (612h11m8.430276309s) [noise=true , address=145.79.227.38:4978 , piece_id=4SL7EQLMSAI4GNHJTEGXFT3TJTNBIIWIBGJK2CLUX2SE5PB3442A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.563806648 (612h11m8.434889693s) [noise=true , address=145.79.227.45:4229 , piece_id=CAWGMIKUIFWQHDFGTGJZWBNEOIIU6NOKNWI5FYDOHIAEI3ZTOUWQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.565585525 (612h11m8.43666857s) [noise=true , address=145.79.227.32:4836 , piece_id=Y2AWRUHQENFG5QPYCHNO5SUZADPKQPAKVKVBOJPQSQ6TUSIF5KMQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.565709672 (612h11m8.436792707s) [noise=true , address=145.79.227.28:4726 , piece_id=G7BHMGKGOUMJIWGFLIDPOFVMDDQEFREOBQ4VCT2MPJO5AGADRZWA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.574173587 (612h11m8.445256632s) [noise=true , address=145.79.227.37:4565 , piece_id=RICSFV3RXT5TKVY2RU7YU7ODVH5YMD653IA6HOMJNY535WILWHZA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.581151284 (612h11m8.452234319s) [noise=true , address=145.79.227.9:4365 , piece_id=FHFKJ5BTRAQBJTARN67A6XBNEM7N7NHC27UMAUV6IULSKA3BGY2A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.588227949 (612h11m8.459310995s) [noise=true , address=145.79.227.14:4838 , piece_id=C2KGOTGETVBAROCJ7E3I4HHULBW4H72D3F7BEKPVUBPKJSBWHXAQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.636319211 (612h11m8.507402256s) [noise=true , address=145.79.227.7:4295 , piece_id=5VZO7W62ISF4AXICMG67HIVXKUQP7UGS2KTFBMB3DUKHY2CM2ONA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.640095251 (612h11m8.511178296s) [noise=true , address=145.79.227.47:4265 , piece_id=AEBKVR5SY6FS6Y7OYZHPJ4TXYTHMAHRTAG6DWMCI25OITOOCI37A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.645671579 (612h11m8.516754654s) [noise=true , address=145.79.227.41:4085 , piece_id=AEMT4DDHEGZWUUKYPW2VH74L3RHCV7X7KNYFPEEU7HHOGU63V5YQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.828391169 (612h11m8.699474204s) [noise=true , address=77.90.31.14:4662 , piece_id=YWOIRX7YBEMUYHNG5IJLD7R5WT4KVM7MZIR4ZDOXSINWZTWCZSZQ, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.906771956 (612h11m8.777854991s) [noise=true , address=45.91.116.11:4118 , piece_id=PO7K2VSEBAP6WXT5X7PY25MTRZOBVDK3LV5YWZIBG5HAL72SWSNA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: <nil>
Dec 31 03:40:08.906790982 (612h11m8.777874017s) [seg_pos={0 0}, upload=536485]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:40:08.906993227 (612h11m8.778076272s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907000601 (612h11m8.778083636s) [noise=true , address=77.90.31.16:4736 , piece_id=FI6ZTHL4TLSFTCOM2BISO6HOMJBZLNZZXSDDKZHBHXSBKKKVGVIA, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907018775 (612h11m8.778101821s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907031459 (612h11m8.778114504s) [noise=true , address=45.91.116.6:4973 , piece_id=2G7OCW57XD6XTQ6ANZPEJYECVKKMEPPY2KZUGYJNIY5ZVWQ2YO4A, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907378109 (612h11m8.778461144s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 31 03:40:08.907385304 (612h11m8.778468339s) [noise=true , address=45.91.116.5:4300 , piece_id=5QLQTJWMMTO6GHQIOCMEG6OA47RUGCUDVRX5ZP6LD5C7TBSUZ6FA, seg_pos={0 0}, upload=536485]: Long tail cancellation due to optimal threshold
Dec 31 03:40:08.907533285 (612h11m8.77861632s) [seg_pos={0 0}, upload=536485]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:40:08.907608989 (612h11m8.778692024s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:40:08.907613688 (612h11m8.778696723s) [upload=536485]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:40:09.204553217 (612h11m9.075636262s) [upload=536485]: Done uploading segments...
Dec 31 03:40:09.204578726 (612h11m9.075661761s) [upload=536485]: Upload finished. err: <nil>
Duration: 30.005528866s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [..X..........X.......X.X...X...X...X.X.X...X......X..XX.....] duration:
3.383676435s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:40:29 UTC)
orbiter warning: context deadline exceeded
Duration: 3.383762449s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
49.876604598s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:40:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 49.8766266s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................X....X.X..X...........] duration:
7.045870879s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:39:29 UTC)
orbiter warning: context deadline exceeded
Duration: 7.045896488s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXXX] duration:
30.003038983s trace:
https://jaeger.dev.storj.io/trace/695fc9e1063e455d
FAIL (2025-12-31 03:39:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: URHFHGLL7732MKNTV77R5OFFC6UIR2OG2AHZAXUCH7IAYKC6C5CQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: ZQJINZG7ME5M3BJVIKIZPPHS4WDUFYRQIHIA6BK4ID32ZMF5N3IA) context deadline exceeded
======LOGS======
170921s) [upload=536469]: Uploading segments...
Dec 31 03:38:54.240739350 (612h9m54.111822385s) [upload=536469]: Got next segment. Inline: false
Dec 31 03:38:54.240778746 (612h9m54.111861781s) [upload=536469]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:38:54.240782854 (612h9m54.111865899s) [upload=536469]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:38:54.485351372 (612h9m54.356434407s) [seg_pos={0 0}, upload=536469]: Begin upload segment...
Dec 31 03:38:54.485399974 (612h9m54.356483019s) [seg_pos={0 0}, upload=536469]: Done begin upload segment.
Dec 31 03:38:54.485449128 (612h9m54.356532163s) [noise=true , address=145.79.227.25:4618 , piece_id=E4QOD4BZALLIW7LCUODVYQUBLZYVVWO4URC3E2FC3DEOV6MXABOA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.485544369 (612h9m54.356627414s) [noise=true , address=77.90.31.14:4662 , piece_id=URHFHGLL7732MKNTV77R5OFFC6UIR2OG2AHZAXUCH7IAYKC6C5CQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.486131266 (612h9m54.357214311s) [noise=true , address=77.90.31.21:4878 , piece_id=CWJEQE3DDKMDR7KD6BHM4RORVTFAIGUJG4XS7NPKP7HFP3AFLSUA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.486595800 (612h9m54.357678846s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQJINZG7ME5M3BJVIKIZPPHS4WDUFYRQIHIA6BK4ID32ZMF5N3IA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.486959985 (612h9m54.35804302s) [noise=true , address=145.79.227.7:4295 , piece_id=H632K5M7LDKQ64UVDI27MKG44DDTY75GI2YKTX4IKGXL3QXR7IGQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.487305593 (612h9m54.358388638s) [noise=true , address=45.91.116.9:4264 , piece_id=WZ7PQRPJBBHAXCETOAKDYUNDJFFLZZRNSQOTGGE5DHRHUANRAVOA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.487645619 (612h9m54.358728664s) [noise=true , address=145.79.227.15:4876 , piece_id=EPJ7R7JHWIIZYZHEMN7YNE4R3VFRGMRTRSPSGZEARKWU3SFG7KZA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.487991648 (612h9m54.359074693s) [noise=true , address=77.90.31.11:4555 , piece_id=5G5AMW53M3LFWHU7WMWZ4YVEZTRN4E7FS6ZE5PHGAX2HZYPTHPDQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.488336826 (612h9m54.359419861s) [noise=true , address=145.79.227.50:4373 , piece_id=DJVVZMW22237OAJZXPORZL7Y2LFN2BAAKQP6KQOMKF3JO64JLGWA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.490061120 (612h9m54.361144165s) [noise=true , address=145.79.227.8:4329 , piece_id=JRJ25NKDMUILTYJ4AJZAGAG4MUBC75HRCNYGWWYZPKADUHCLLLKQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.490563696 (612h9m54.361646731s) [noise=true , address=145.79.227.37:4565 , piece_id=EDO3UME5HD5V7YQCZAIQIPFRE7WANYNFHH3G2IGL666L5L7SITFQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.490924212 (612h9m54.362007247s) [noise=true , address=145.79.227.21:4473 , piece_id=XZRRIYOPLBKCJK3JHUBPG6XOFHSNTVQDTEU37QUQ34FGQCL57HJQ, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.491263458 (612h9m54.362346493s) [noise=true , address=145.79.227.41:4087 , piece_id=HTBWCLUIEDKNXZ75LUAVPQ72ZHHZLBDBPJ5HA6PQN4GGVFQYGRCA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.491643131 (612h9m54.362726177s) [noise=true , address=145.79.227.38:4978 , piece_id=7PYBUQQ4XKEOWYMYHGR2ZCVMXWUBIZCLGGCKF47PUKIOU5ATR6HA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.491971216 (612h9m54.363054251s) [noise=true , address=145.79.227.23:4546 , piece_id=3NXXRABPVZEICHMF3YSOLURCI2GXYQANYTGYBHAH5UU27VV7MTIA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:54.492324439 (612h9m54.363407474s) [noise=true , address=145.79.227.17:4972 , piece_id=4OBPN6434BGCL5B2KGDSFXNAK7JBMQOQXBRNNGMON756Y5BLUYYA, seg_pos={0 0}, upload=536469]: Uploading piece...
Dec 31 03:38:56.162084934 (612h9m56.033167969s) [upload=536469]: Next returned nil segment
Dec 31 03:38:56.162097508 (612h9m56.033180543s) [upload=536469]: Waiting for error group managing segments...
Dec 31 03:38:56.460629018 (612h9m56.331712063s) [noise=true , address=145.79.227.25:4618 , piece_id=E4QOD4BZALLIW7LCUODVYQUBLZYVVWO4URC3E2FC3DEOV6MXABOA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.467179742 (612h9m56.338262787s) [noise=true , address=145.79.227.37:4565 , piece_id=EDO3UME5HD5V7YQCZAIQIPFRE7WANYNFHH3G2IGL666L5L7SITFQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.475738609 (612h9m56.346821644s) [noise=true , address=145.79.227.8:4329 , piece_id=JRJ25NKDMUILTYJ4AJZAGAG4MUBC75HRCNYGWWYZPKADUHCLLLKQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.477502978 (612h9m56.348586023s) [noise=true , address=145.79.227.38:4978 , piece_id=7PYBUQQ4XKEOWYMYHGR2ZCVMXWUBIZCLGGCKF47PUKIOU5ATR6HA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.487900777 (612h9m56.358983822s) [noise=true , address=145.79.227.7:4295 , piece_id=H632K5M7LDKQ64UVDI27MKG44DDTY75GI2YKTX4IKGXL3QXR7IGQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.487964669 (612h9m56.359047714s) [noise=true , address=145.79.227.15:4876 , piece_id=EPJ7R7JHWIIZYZHEMN7YNE4R3VFRGMRTRSPSGZEARKWU3SFG7KZA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.495867287 (612h9m56.366950332s) [noise=true , address=145.79.227.17:4972 , piece_id=4OBPN6434BGCL5B2KGDSFXNAK7JBMQOQXBRNNGMON756Y5BLUYYA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.565731566 (612h9m56.436814601s) [noise=true , address=145.79.227.21:4473 , piece_id=XZRRIYOPLBKCJK3JHUBPG6XOFHSNTVQDTEU37QUQ34FGQCL57HJQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.567193690 (612h9m56.438276725s) [noise=true , address=145.79.227.23:4546 , piece_id=3NXXRABPVZEICHMF3YSOLURCI2GXYQANYTGYBHAH5UU27VV7MTIA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.664657053 (612h9m56.535740098s) [noise=true , address=77.90.31.14:4662 , piece_id=URHFHGLL7732MKNTV77R5OFFC6UIR2OG2AHZAXUCH7IAYKC6C5CQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:56.753291864 (612h9m56.6243749s) [noise=true , address=77.90.31.21:4878 , piece_id=CWJEQE3DDKMDR7KD6BHM4RORVTFAIGUJG4XS7NPKP7HFP3AFLSUA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:57.029104947 (612h9m56.900187992s) [noise=true , address=45.91.116.9:4264 , piece_id=WZ7PQRPJBBHAXCETOAKDYUNDJFFLZZRNSQOTGGE5DHRHUANRAVOA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:57.150202962 (612h9m57.021285997s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQJINZG7ME5M3BJVIKIZPPHS4WDUFYRQIHIA6BK4ID32ZMF5N3IA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: <nil>
Dec 31 03:38:57.150222389 (612h9m57.021305424s) [seg_pos={0 0}, upload=536469]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:38:57.150450994 (612h9m57.021534029s) [noise=true , address=145.79.227.50:4373 , piece_id=DJVVZMW22237OAJZXPORZL7Y2LFN2BAAKQP6KQOMKF3JO64JLGWA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:55774->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:55774->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55774->145.79.227.50:4373: use of closed network connection
Dec 31 03:38:57.150457436 (612h9m57.021540481s) [noise=true , address=145.79.227.50:4373 , piece_id=DJVVZMW22237OAJZXPORZL7Y2LFN2BAAKQP6KQOMKF3JO64JLGWA, seg_pos={0 0}, upload=536469]: Long tail cancellation due to optimal threshold
Dec 31 03:38:57.150653650 (612h9m57.021736685s) [noise=true , address=145.79.227.41:4087 , piece_id=HTBWCLUIEDKNXZ75LUAVPQ72ZHHZLBDBPJ5HA6PQN4GGVFQYGRCA, seg_pos={0 0}, upload=536469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:54000->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:54000->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54000->145.79.227.41:4087: use of closed network connection
Dec 31 03:38:57.150661124 (612h9m57.021744169s) [noise=true , address=145.79.227.41:4087 , piece_id=HTBWCLUIEDKNXZ75LUAVPQ72ZHHZLBDBPJ5HA6PQN4GGVFQYGRCA, seg_pos={0 0}, upload=536469]: Long tail cancellation due to optimal threshold
Dec 31 03:38:57.150755083 (612h9m57.021838118s) [noise=true , address=77.90.31.11:4555 , piece_id=5G5AMW53M3LFWHU7WMWZ4YVEZTRN4E7FS6ZE5PHGAX2HZYPTHPDQ, seg_pos={0 0}, upload=536469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 31 03:38:57.150760103 (612h9m57.021843138s) [noise=true , address=77.90.31.11:4555 , piece_id=5G5AMW53M3LFWHU7WMWZ4YVEZTRN4E7FS6ZE5PHGAX2HZYPTHPDQ, seg_pos={0 0}, upload=536469]: Long tail cancellation due to optimal threshold
Dec 31 03:38:57.150775642 (612h9m57.021858677s) [seg_pos={0 0}, upload=536469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:38:57.150837550 (612h9m57.021920585s) [upload=536469]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:38:57.150842349 (612h9m57.021925394s) [upload=536469]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:38:57.436027967 (612h9m57.307111002s) [upload=536469]: Done uploading segments...
Dec 31 03:38:57.436050360 (612h9m57.307133395s) [upload=536469]: Upload finished. err: <nil>
Duration: 30.003057138s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
41.659414227s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:39:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 41.65943174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...X..........X.......X.X...X...X...X.X.X...X......X..XX....] duration:
3.490424672s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:39:20 UTC)
orbiter warning: context deadline exceeded
Duration: 3.490452805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X....X.X..X..........] duration:
5.835412241s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:38:52 UTC)
orbiter warning: context deadline exceeded
Duration: 5.835438912s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XXX] duration:
30.003791303s trace:
https://jaeger.dev.storj.io/trace/79d6658ef834ea26
FAIL (2025-12-31 03:38:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6461]: Starting upload
Dec 31 03:38:19.534402023 (612h9m19.405485068s) [upload=536461]: Done starting upload
Dec 31 03:38:19.534511591 (612h9m19.405594626s) [upload=536461]: Uploading segments...
Dec 31 03:38:19.534601563 (612h9m19.405684608s) [upload=536461]: Got next segment. Inline: false
Dec 31 03:38:19.534662218 (612h9m19.405745263s) [upload=536461]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:38:19.534669753 (612h9m19.405752788s) [upload=536461]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:38:19.780438564 (612h9m19.651521599s) [seg_pos={0 0}, upload=536461]: Begin upload segment...
Dec 31 03:38:19.780500001 (612h9m19.651583046s) [seg_pos={0 0}, upload=536461]: Done begin upload segment.
Dec 31 03:38:19.780572038 (612h9m19.651655073s) [noise=true , address=145.79.227.36:4519 , piece_id=2MP4CXKOE756XZ2SWSHY4RPR7LN67ISSFQGJ6R4X3SVR74KFIJEQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.780653333 (612h9m19.651736388s) [noise=true , address=145.79.227.20:4439 , piece_id=6WOQVGVBVOCKQMJRVOPY5CMZSLAMQV3R5KWRWZVW7TJTU6TD5MDA, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.781184404 (612h9m19.652267449s) [noise=true , address=45.91.116.11:4118 , piece_id=R2HGBBMCZOZN3QLPFNTC7RTVBSTSLE3VINYDUHBQJ2PNMC6WDAMQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.781663957 (612h9m19.652747002s) [noise=true , address=145.79.227.18:4006 , piece_id=QYCDN6Z5PPNX6IU42Q7B3T3KBUM7M4AYRMTMBG36WNJ5JFLRIGEA, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.782131878 (612h9m19.653214923s) [noise=true , address=45.91.116.8:4048 , piece_id=6PQXD4ZIYFGPPXDALNF2FSKTOHKJPJFMTX764ZDV5LTSX572RUJA, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.782763631 (612h9m19.653846667s) [noise=true , address=77.90.31.10:4518 , piece_id=MPJZBQWSIR7AFXV3PFUWQ2V3ZAEYRPIVCEHKLEFVMGCWV3ZQAO6Q, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.784119362 (612h9m19.655202397s) [noise=true , address=145.79.227.50:4373 , piece_id=7SHSWJT73CC574NNBCS54SX2HQNFGGYSJWMWNIFWAJI75XRPL5DQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.784461253 (612h9m19.655544298s) [noise=true , address=145.79.227.42:4123 , piece_id=P5SVZCYJH7K7YRFBS4CB7SWVREYJCBCKH2EPLO7KLPE5WODQPEXQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.785206302 (612h9m19.656289347s) [noise=true , address=145.79.227.26:4654 , piece_id=DWNANHSE7B3IMYNH665MA5JJB7BQX45OVTQKN43V7AG7ETA7YE4A, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.786410705 (612h9m19.65749374s) [noise=true , address=145.79.227.10:4401 , piece_id=XVT2QXXKPQWNIKXFSFTU6GASTI5BPEVVBN3JQAVWGUPIZE7AJ67A, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.786556443 (612h9m19.657639478s) [noise=true , address=45.91.116.9:4262 , piece_id=JN5MP5GUH66PDH3A4T6UCNBXBKJVCQWOMFFOHGGYHB4IBFZGUHRA, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.788073933 (612h9m19.659156968s) [noise=true , address=77.90.31.19:4842 , piece_id=SMDPJO2BKPUPKBDHLHOEKRRJW6GGMDHERAW54BBJFROSLI57FTVQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.788375095 (612h9m19.659458141s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYYRXJAWEEU4GLFZNLUKKTOFANWJ5RMJ36VUHU2UP3YK32G2VOKQ, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.789538882 (612h9m19.660621917s) [noise=true , address=145.79.227.29:4943 , piece_id=MTGS5ZPFDYMVBLQFSBAVHWR7RL25YSFE4BZ2LF6DB2I44GYR4V5Q, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.790691647 (612h9m19.661774692s) [noise=true , address=45.91.116.7:4011 , piece_id=EFOO6AXTBTMQNMBMFS342MUVEJDJJZME5PQCK6TQRJSCINHDOKDA, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:19.792921743 (612h9m19.664004789s) [noise=true , address=145.79.227.44:4194 , piece_id=DSSZ3S5SN5KKAASXIHDC354D5VQOSVU7OEZVSBCJML34K4JO5Q6A, seg_pos={0 0}, upload=536461]: Uploading piece...
Dec 31 03:38:21.385788603 (612h9m21.256871648s) [upload=536461]: Next returned nil segment
Dec 31 03:38:21.385912008 (612h9m21.256995083s) [upload=536461]: Waiting for error group managing segments...
Dec 31 03:38:21.767609577 (612h9m21.638692612s) [noise=true , address=145.79.227.12:4765 , piece_id=ZYYRXJAWEEU4GLFZNLUKKTOFANWJ5RMJ36VUHU2UP3YK32G2VOKQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.776782453 (612h9m21.647865498s) [noise=true , address=145.79.227.36:4519 , piece_id=2MP4CXKOE756XZ2SWSHY4RPR7LN67ISSFQGJ6R4X3SVR74KFIJEQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.777414627 (612h9m21.648497662s) [noise=true , address=145.79.227.26:4654 , piece_id=DWNANHSE7B3IMYNH665MA5JJB7BQX45OVTQKN43V7AG7ETA7YE4A, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.778297057 (612h9m21.649380092s) [noise=true , address=145.79.227.50:4373 , piece_id=7SHSWJT73CC574NNBCS54SX2HQNFGGYSJWMWNIFWAJI75XRPL5DQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.778415132 (612h9m21.649498187s) [noise=true , address=145.79.227.20:4439 , piece_id=6WOQVGVBVOCKQMJRVOPY5CMZSLAMQV3R5KWRWZVW7TJTU6TD5MDA, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.797873954 (612h9m21.668956989s) [noise=true , address=145.79.227.10:4401 , piece_id=XVT2QXXKPQWNIKXFSFTU6GASTI5BPEVVBN3JQAVWGUPIZE7AJ67A, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.857567036 (612h9m21.728650071s) [noise=true , address=145.79.227.18:4006 , piece_id=QYCDN6Z5PPNX6IU42Q7B3T3KBUM7M4AYRMTMBG36WNJ5JFLRIGEA, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.869030043 (612h9m21.740113078s) [noise=true , address=145.79.227.42:4123 , piece_id=P5SVZCYJH7K7YRFBS4CB7SWVREYJCBCKH2EPLO7KLPE5WODQPEXQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:21.992285698 (612h9m21.863368743s) [noise=true , address=145.79.227.29:4943 , piece_id=MTGS5ZPFDYMVBLQFSBAVHWR7RL25YSFE4BZ2LF6DB2I44GYR4V5Q, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:22.061510940 (612h9m21.932593985s) [noise=true , address=77.90.31.19:4842 , piece_id=SMDPJO2BKPUPKBDHLHOEKRRJW6GGMDHERAW54BBJFROSLI57FTVQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:22.095641669 (612h9m21.966724724s) [noise=true , address=45.91.116.8:4048 , piece_id=6PQXD4ZIYFGPPXDALNF2FSKTOHKJPJFMTX764ZDV5LTSX572RUJA, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:22.149244945 (612h9m22.02032799s) [noise=true , address=45.91.116.7:4011 , piece_id=EFOO6AXTBTMQNMBMFS342MUVEJDJJZME5PQCK6TQRJSCINHDOKDA, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:22.202560685 (612h9m22.07364373s) [noise=true , address=45.91.116.11:4118 , piece_id=R2HGBBMCZOZN3QLPFNTC7RTVBSTSLE3VINYDUHBQJ2PNMC6WDAMQ, seg_pos={0 0}, upload=536461]: Done uploading piece. err: <nil>
Dec 31 03:38:22.202584351 (612h9m22.073667396s) [seg_pos={0 0}, upload=536461]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:38:22.202793549 (612h9m22.073876584s) [noise=true , address=145.79.227.44:4194 , piece_id=DSSZ3S5SN5KKAASXIHDC354D5VQOSVU7OEZVSBCJML34K4JO5Q6A, seg_pos={0 0}, upload=536461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 31 03:38:22.202800432 (612h9m22.073883467s) [noise=true , address=145.79.227.44:4194 , piece_id=DSSZ3S5SN5KKAASXIHDC354D5VQOSVU7OEZVSBCJML34K4JO5Q6A, seg_pos={0 0}, upload=536461]: Long tail cancellation due to optimal threshold
Dec 31 03:38:22.202900352 (612h9m22.073983387s) [noise=true , address=45.91.116.9:4262 , piece_id=JN5MP5GUH66PDH3A4T6UCNBXBKJVCQWOMFFOHGGYHB4IBFZGUHRA, seg_pos={0 0}, upload=536461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:59982->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:59982->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59982->45.91.116.9:4262: use of closed network connection
Dec 31 03:38:22.202905462 (612h9m22.073988497s) [noise=true , address=45.91.116.9:4262 , piece_id=JN5MP5GUH66PDH3A4T6UCNBXBKJVCQWOMFFOHGGYHB4IBFZGUHRA, seg_pos={0 0}, upload=536461]: Long tail cancellation due to optimal threshold
Dec 31 03:38:22.202978811 (612h9m22.074061866s) [noise=true , address=77.90.31.10:4518 , piece_id=MPJZBQWSIR7AFXV3PFUWQ2V3ZAEYRPIVCEHKLEFVMGCWV3ZQAO6Q, seg_pos={0 0}, upload=536461]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:52716->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 03:38:22.202989451 (612h9m22.074072486s) [noise=true , address=77.90.31.10:4518 , piece_id=MPJZBQWSIR7AFXV3PFUWQ2V3ZAEYRPIVCEHKLEFVMGCWV3ZQAO6Q, seg_pos={0 0}, upload=536461]: Long tail cancellation due to optimal threshold
Dec 31 03:38:22.203023186 (612h9m22.074106221s) [seg_pos={0 0}, upload=536461]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:38:22.203111103 (612h9m22.074194138s) [upload=536461]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:38:22.203116914 (612h9m22.074199949s) [upload=536461]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:38:22.489691246 (612h9m22.360774281s) [upload=536461]: Done uploading segments...
Dec 31 03:38:22.489709119 (612h9m22.360792155s) [upload=536461]: Upload finished. err: <nil>
Duration: 30.00380525s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
38.7262342s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:38:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 38.726252655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [....X..........X.......X.X...X...X...X.X.X...X......X..XX...] duration:
3.695825325s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:38:36 UTC)
orbiter warning: context deadline exceeded
Duration: 3.695850764s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................X....X.X..X.........] duration:
6.374656813s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:38:22 UTC)
orbiter warning: context deadline exceeded
Duration: 6.374684757s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.X...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.XX] duration:
30.002451447s trace:
https://jaeger.dev.storj.io/trace/6628a3fcec699e97
FAIL (2025-12-31 03:38:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 03:37:49.301644986 (612h8m49.172728021s) [upload=536453]: Starting upload
Dec 31 03:37:49.301693678 (612h8m49.172776713s) [upload=536453]: Done starting upload
Dec 31 03:37:49.301886494 (612h8m49.17296954s) [upload=536453]: Uploading segments...
Dec 31 03:37:49.301925059 (612h8m49.173008094s) [upload=536453]: Got next segment. Inline: false
Dec 31 03:37:49.301949154 (612h8m49.173032189s) [upload=536453]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:37:49.301952911 (612h8m49.173035946s) [upload=536453]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:37:49.543924415 (612h8m49.41500745s) [seg_pos={0 0}, upload=536453]: Begin upload segment...
Dec 31 03:37:49.543980992 (612h8m49.415064027s) [seg_pos={0 0}, upload=536453]: Done begin upload segment.
Dec 31 03:37:49.544032991 (612h8m49.415116026s) [noise=true , address=145.79.227.37:4567 , piece_id=DZ744I4UW3B7ANS7OYKPTGD4QPZJZ5KKOU3L77SZBPY5HHONOXNQ, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.544039373 (612h8m49.415122408s) [noise=true , address=145.79.227.17:4970 , piece_id=7ZPQ62CMMIJRTJU7IZ4UJXU6E2YBC6U2OKF5CTGBTZXZ6CBBHEDA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.544635248 (612h8m49.415718293s) [noise=true , address=45.91.116.8:4046 , piece_id=ZJXMN7HMLW5FKR5XTZMFN6AKIEGQQIJWVCJSB4TAEMWMUDUUWERQ, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.545029608 (612h8m49.416112643s) [noise=true , address=77.90.31.10:4520 , piece_id=KW6W7CZT6JVKQLBRUJIHFZ3GKWDHMFZBW4Y546P4WJXNIO6MLDYQ, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.545389935 (612h8m49.41647297s) [noise=true , address=45.91.116.12:4227 , piece_id=UQDIRLZJYSFDF5B2HIPRDZLYU5P3NADBDY3HFUNPZ4GC32VR4ATA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.546556756 (612h8m49.417639791s) [noise=true , address=145.79.227.26:4655 , piece_id=YTVW7TFHQNCWCEGXY2JR3EQJS32LPR6CIZEHNPT3CDIIIHBIIFZQ, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.547827487 (612h8m49.418910532s) [noise=true , address=145.79.227.7:4295 , piece_id=5DPB7Q6XZEFL37PHSQ5Z23VZ2K7E5ESZ3DSI6PQBSQPFGA2YM76A, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.548396349 (612h8m49.419479384s) [noise=true , address=45.91.116.14:4191 , piece_id=D4TDLZQT42STDM7SX6T63PUJEY5ER5RCDNOLEUCG5KKFP6AGPJ5Q, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.549649516 (612h8m49.420732551s) [noise=true , address=145.79.227.19:4078 , piece_id=AEB7BDQV5N5DN4IVQMLNQDRMWCPNZDUXLDTHVOTMH3YCY3P72SWA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.550849270 (612h8m49.421932315s) [noise=true , address=145.79.227.32:4835 , piece_id=U3YTRPLTQBKPKSRJUFUZFIV54EUFBZ6PWJ6QJKCIYUQNU5WXRJRA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.552116753 (612h8m49.423199798s) [noise=true , address=145.79.227.23:4547 , piece_id=OZG4ND5BD6Z2JYLFPJGNOTXA5ZJKJL2LDFCGXABGRJSF3V3XF2TA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.553293404 (612h8m49.424376449s) [noise=true , address=145.79.227.49:4338 , piece_id=DNFCCKO6B5OZWRIDXRMI3EBFZ7SIJK4URB7BYEPYQ2RN3Z35Z7LA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.555434661 (612h8m49.426517696s) [noise=true , address=145.79.227.6:4259 , piece_id=RJ7RN46JWHMASRYV2KJARMIOPFJOA3CZSPGMBCRRAT2VZZESTH2Q, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.555738950 (612h8m49.426821995s) [noise=true , address=77.90.31.13:4627 , piece_id=54TIOOKT3TLTZMUALFAWW6WJ77R2RWFGXAAG4LU2WCUK34DSOAJA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.558041404 (612h8m49.429124449s) [noise=true , address=145.79.227.51:4409 , piece_id=ESGIJMJMYXOIV3354VHRURFDLQLHN7IM677JVAYRLPTMP5R77PEA, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:49.558792234 (612h8m49.429875299s) [noise=true , address=145.79.227.13:4803 , piece_id=NDSXXABKTZ3L23AUNEFY4LV3CALLRZ6RL7EELIQS3RYR645LO52A, seg_pos={0 0}, upload=536453]: Uploading piece...
Dec 31 03:37:51.152164839 (612h8m51.023247884s) [upload=536453]: Next returned nil segment
Dec 31 03:37:51.152179457 (612h8m51.023262502s) [upload=536453]: Waiting for error group managing segments...
Dec 31 03:37:51.527267941 (612h8m51.398350986s) [noise=true , address=145.79.227.23:4547 , piece_id=OZG4ND5BD6Z2JYLFPJGNOTXA5ZJKJL2LDFCGXABGRJSF3V3XF2TA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.533216036 (612h8m51.404299072s) [noise=true , address=145.79.227.19:4078 , piece_id=AEB7BDQV5N5DN4IVQMLNQDRMWCPNZDUXLDTHVOTMH3YCY3P72SWA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.535448036 (612h8m51.406531082s) [noise=true , address=145.79.227.26:4655 , piece_id=YTVW7TFHQNCWCEGXY2JR3EQJS32LPR6CIZEHNPT3CDIIIHBIIFZQ, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.537314500 (612h8m51.408397545s) [noise=true , address=145.79.227.7:4295 , piece_id=5DPB7Q6XZEFL37PHSQ5Z23VZ2K7E5ESZ3DSI6PQBSQPFGA2YM76A, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.537711456 (612h8m51.408794501s) [noise=true , address=145.79.227.13:4803 , piece_id=NDSXXABKTZ3L23AUNEFY4LV3CALLRZ6RL7EELIQS3RYR645LO52A, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.557232788 (612h8m51.428315823s) [noise=true , address=145.79.227.32:4835 , piece_id=U3YTRPLTQBKPKSRJUFUZFIV54EUFBZ6PWJ6QJKCIYUQNU5WXRJRA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.559117126 (612h8m51.430200161s) [noise=true , address=145.79.227.37:4567 , piece_id=DZ744I4UW3B7ANS7OYKPTGD4QPZJZ5KKOU3L77SZBPY5HHONOXNQ, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.561940592 (612h8m51.433023627s) [noise=true , address=145.79.227.51:4409 , piece_id=ESGIJMJMYXOIV3354VHRURFDLQLHN7IM677JVAYRLPTMP5R77PEA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.628465773 (612h8m51.499548808s) [noise=true , address=145.79.227.6:4259 , piece_id=RJ7RN46JWHMASRYV2KJARMIOPFJOA3CZSPGMBCRRAT2VZZESTH2Q, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.827548541 (612h8m51.698631586s) [noise=true , address=77.90.31.10:4520 , piece_id=KW6W7CZT6JVKQLBRUJIHFZ3GKWDHMFZBW4Y546P4WJXNIO6MLDYQ, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:51.887665301 (612h8m51.758748336s) [noise=true , address=45.91.116.8:4046 , piece_id=ZJXMN7HMLW5FKR5XTZMFN6AKIEGQQIJWVCJSB4TAEMWMUDUUWERQ, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:52.194269292 (612h8m52.065352337s) [noise=true , address=45.91.116.12:4227 , piece_id=UQDIRLZJYSFDF5B2HIPRDZLYU5P3NADBDY3HFUNPZ4GC32VR4ATA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:52.979811009 (612h8m52.850894044s) [noise=true , address=145.79.227.49:4338 , piece_id=DNFCCKO6B5OZWRIDXRMI3EBFZ7SIJK4URB7BYEPYQ2RN3Z35Z7LA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: <nil>
Dec 31 03:37:52.979831097 (612h8m52.850914132s) [seg_pos={0 0}, upload=536453]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:37:52.980037540 (612h8m52.851120585s) [noise=true , address=77.90.31.13:4627 , piece_id=54TIOOKT3TLTZMUALFAWW6WJ77R2RWFGXAAG4LU2WCUK34DSOAJA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 31 03:37:52.980045014 (612h8m52.851128059s) [noise=true , address=77.90.31.13:4627 , piece_id=54TIOOKT3TLTZMUALFAWW6WJ77R2RWFGXAAG4LU2WCUK34DSOAJA, seg_pos={0 0}, upload=536453]: Long tail cancellation due to optimal threshold
Dec 31 03:37:52.980106191 (612h8m52.851189226s) [noise=true , address=145.79.227.17:4970 , piece_id=7ZPQ62CMMIJRTJU7IZ4UJXU6E2YBC6U2OKF5CTGBTZXZ6CBBHEDA, seg_pos={0 0}, upload=536453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: expected piece hash; context canceled; EOF
Dec 31 03:37:52.980125879 (612h8m52.851208924s) [noise=true , address=145.79.227.17:4970 , piece_id=7ZPQ62CMMIJRTJU7IZ4UJXU6E2YBC6U2OKF5CTGBTZXZ6CBBHEDA, seg_pos={0 0}, upload=536453]: Long tail cancellation due to optimal threshold
Dec 31 03:37:52.980181624 (612h8m52.851264659s) [noise=true , address=45.91.116.14:4191 , piece_id=D4TDLZQT42STDM7SX6T63PUJEY5ER5RCDNOLEUCG5KKFP6AGPJ5Q, seg_pos={0 0}, upload=536453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 31 03:37:52.980188448 (612h8m52.851271483s) [noise=true , address=45.91.116.14:4191 , piece_id=D4TDLZQT42STDM7SX6T63PUJEY5ER5RCDNOLEUCG5KKFP6AGPJ5Q, seg_pos={0 0}, upload=536453]: Long tail cancellation due to optimal threshold
Dec 31 03:37:52.980203637 (612h8m52.851286672s) [seg_pos={0 0}, upload=536453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:37:52.980272999 (612h8m52.851356044s) [upload=536453]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:37:52.980277597 (612h8m52.851360632s) [upload=536453]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:37:53.262944090 (612h8m53.134027125s) [upload=536453]: Done uploading segments...
Dec 31 03:37:53.262962725 (612h8m53.13404576s) [upload=536453]: Upload finished. err: <nil>
Duration: 30.002475082s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.....X..........X.......X.X...X...X...X.X.X...X......X..XX..] duration:
3.617987338s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:38:16 UTC)
orbiter warning: context deadline exceeded
Duration: 3.618015161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X....X.X..X........] duration:
8.841964832s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:38:09 UTC)
orbiter warning: context deadline exceeded
Duration: 8.841994809s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
45.361373357s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:38:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 45.361388848s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [......X..........X.......X.X...X...X...X.X.X...X......X..XX.] duration:
3.566893127s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:37:48 UTC)
orbiter warning: context deadline exceeded
Duration: 3.566919176s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XX.X...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.X] duration:
30.003284524s trace:
https://jaeger.dev.storj.io/trace/4504a91b29eb79e5
FAIL (2025-12-31 03:37:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 03:37:17.651761437 (612h8m17.522844472s) [upload=536446]: Starting upload
Dec 31 03:37:17.651833023 (612h8m17.522916058s) [upload=536446]: Done starting upload
Dec 31 03:37:17.652024799 (612h8m17.523107844s) [upload=536446]: Uploading segments...
Dec 31 03:37:17.652063523 (612h8m17.523146568s) [upload=536446]: Got next segment. Inline: false
Dec 31 03:37:17.652090163 (612h8m17.523173198s) [upload=536446]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:37:17.652094301 (612h8m17.523177336s) [upload=536446]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:37:17.898228680 (612h8m17.769311725s) [seg_pos={0 0}, upload=536446]: Begin upload segment...
Dec 31 03:37:17.898283535 (612h8m17.76936658s) [seg_pos={0 0}, upload=536446]: Done begin upload segment.
Dec 31 03:37:17.898307671 (612h8m17.769390706s) [noise=true , address=145.79.227.40:4049 , piece_id=5ICAF24EEJMQ7F75JYZCOHQ5GYTWOV2VYFROBIFAMS4SGPYM2P5A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.898408362 (612h8m17.769491427s) [noise=true , address=45.91.116.11:4119 , piece_id=LMPWFFARZ7ZHSUDWH3WS5IRMTBDEVNAD6CWELPAYWMD6KXUNQQWQ, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.898509094 (612h8m17.769592129s) [noise=true , address=145.79.227.37:4567 , piece_id=FHC26WAVVR5NJVMQCBMD5GLX26G2LVOCEGCQRRRHB3NA4HTMPQBA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.899130047 (612h8m17.770213092s) [noise=true , address=77.90.31.21:4880 , piece_id=XEW25HZ4KNNYM5UEUA5XKALF3LLIPBIKGPHOTGFRBE3K7E67S53A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.899587197 (612h8m17.770670242s) [noise=true , address=145.79.227.32:4836 , piece_id=I6KGLMX3FPAXTZ4GOJRAPUEWJ5EYUYQBSJ7WI7E4R7GOYHMINW3A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.899960749 (612h8m17.771043784s) [noise=true , address=145.79.227.21:4473 , piece_id=23WJ6XMXDMAJ5MAGYSVFV7Q36LVIN7RZJKQJJJYACRSNVND3VSZA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.900305956 (612h8m17.771388991s) [noise=true , address=77.90.31.17:4770 , piece_id=KFQ74B3EMUDYXDJ34VIOFB3QNLDTGLDHE536DON25DEQKNFGTS2A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.900725876 (612h8m17.771808921s) [noise=true , address=145.79.227.7:4293 , piece_id=K32MRMOJKLPHD7UERAHRG72NY3E5MD6KDUK26ODOF2W2N7X4DSTA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.901145394 (612h8m17.772228429s) [noise=true , address=45.91.116.13:4153 , piece_id=LFZ6TY73OHQZ5H5HS4FYUWASIFNY465RMRJNGHGHR5KLK3ZET2MA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.901464782 (612h8m17.772547817s) [noise=true , address=145.79.227.9:4367 , piece_id=BW3P5Z6PDDLEGGE7ZW3NFX7YV43QM55RSZYP3DYGLAHFAABJKGGA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.901732532 (612h8m17.772815568s) [noise=true , address=145.79.227.31:4797 , piece_id=GMHUOYIS7U4WA2EQECY5ZO6V6NBFYVPVM6EZF6P34VQ3CZKUPL7A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.902001625 (612h8m17.77308466s) [noise=true , address=145.79.227.24:4582 , piece_id=HXJZBMLMVWMRJMJFWTZ5YN22HUE7YF47VAPUIJB7DLPXEHZ5GQ2A, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.902059304 (612h8m17.773142339s) [noise=true , address=145.79.227.6:4257 , piece_id=VYSA54ERESFCHCUZ4DBU27P5VPISGH7BHHQ7EOO4MBBNP2CHQBKQ, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.902485086 (612h8m17.773568121s) [noise=true , address=145.79.227.10:4403 , piece_id=I3PIN7EUISAIHWB2JTXTJDWORYGZ22JHYOEG36YMXUC7RUBE7FHQ, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.902803131 (612h8m17.773886166s) [noise=true , address=145.79.227.8:4331 , piece_id=5NKCOCK4J3URZHSPVKAE7AIGV7CWSLFKIQMVKC37FEWVRUZ2W6SA, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:17.903081962 (612h8m17.774164997s) [noise=true , address=145.79.227.2:4113 , piece_id=5XX5NG6D67L4EN2HOA4YXUYYMRNN4PPNGFKC4HLNZDNKNDCUOQNQ, seg_pos={0 0}, upload=536446]: Uploading piece...
Dec 31 03:37:19.502967839 (612h8m19.374050884s) [upload=536446]: Next returned nil segment
Dec 31 03:37:19.502983568 (612h8m19.374066603s) [upload=536446]: Waiting for error group managing segments...
Dec 31 03:37:19.882307667 (612h8m19.753390712s) [noise=true , address=145.79.227.7:4293 , piece_id=K32MRMOJKLPHD7UERAHRG72NY3E5MD6KDUK26ODOF2W2N7X4DSTA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.883435705 (612h8m19.75451875s) [noise=true , address=145.79.227.2:4113 , piece_id=5XX5NG6D67L4EN2HOA4YXUYYMRNN4PPNGFKC4HLNZDNKNDCUOQNQ, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.884401404 (612h8m19.755484439s) [noise=true , address=145.79.227.8:4331 , piece_id=5NKCOCK4J3URZHSPVKAE7AIGV7CWSLFKIQMVKC37FEWVRUZ2W6SA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.884639307 (612h8m19.755722342s) [noise=true , address=145.79.227.32:4836 , piece_id=I6KGLMX3FPAXTZ4GOJRAPUEWJ5EYUYQBSJ7WI7E4R7GOYHMINW3A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.886823737 (612h8m19.757906782s) [noise=true , address=145.79.227.31:4797 , piece_id=GMHUOYIS7U4WA2EQECY5ZO6V6NBFYVPVM6EZF6P34VQ3CZKUPL7A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.887485096 (612h8m19.758568141s) [noise=true , address=145.79.227.6:4257 , piece_id=VYSA54ERESFCHCUZ4DBU27P5VPISGH7BHHQ7EOO4MBBNP2CHQBKQ, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.889026751 (612h8m19.760109786s) [noise=true , address=145.79.227.10:4403 , piece_id=I3PIN7EUISAIHWB2JTXTJDWORYGZ22JHYOEG36YMXUC7RUBE7FHQ, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.893408836 (612h8m19.764491881s) [noise=true , address=145.79.227.9:4367 , piece_id=BW3P5Z6PDDLEGGE7ZW3NFX7YV43QM55RSZYP3DYGLAHFAABJKGGA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.911907259 (612h8m19.782990304s) [noise=true , address=145.79.227.21:4473 , piece_id=23WJ6XMXDMAJ5MAGYSVFV7Q36LVIN7RZJKQJJJYACRSNVND3VSZA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:19.913087065 (612h8m19.7841701s) [noise=true , address=145.79.227.24:4582 , piece_id=HXJZBMLMVWMRJMJFWTZ5YN22HUE7YF47VAPUIJB7DLPXEHZ5GQ2A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:20.184237057 (612h8m20.055320102s) [noise=true , address=77.90.31.21:4880 , piece_id=XEW25HZ4KNNYM5UEUA5XKALF3LLIPBIKGPHOTGFRBE3K7E67S53A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:20.211503306 (612h8m20.082586351s) [noise=true , address=77.90.31.17:4770 , piece_id=KFQ74B3EMUDYXDJ34VIOFB3QNLDTGLDHE536DON25DEQKNFGTS2A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:20.325000589 (612h8m20.196083634s) [noise=true , address=45.91.116.11:4119 , piece_id=LMPWFFARZ7ZHSUDWH3WS5IRMTBDEVNAD6CWELPAYWMD6KXUNQQWQ, seg_pos={0 0}, upload=536446]: Done uploading piece. err: <nil>
Dec 31 03:37:20.325020115 (612h8m20.19610315s) [seg_pos={0 0}, upload=536446]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:37:20.325234463 (612h8m20.196317508s) [noise=true , address=45.91.116.13:4153 , piece_id=LFZ6TY73OHQZ5H5HS4FYUWASIFNY465RMRJNGHGHR5KLK3ZET2MA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 31 03:37:20.325242860 (612h8m20.196325895s) [noise=true , address=45.91.116.13:4153 , piece_id=LFZ6TY73OHQZ5H5HS4FYUWASIFNY465RMRJNGHGHR5KLK3ZET2MA, seg_pos={0 0}, upload=536446]: Long tail cancellation due to optimal threshold
Dec 31 03:37:20.325350174 (612h8m20.196433209s) [noise=true , address=145.79.227.40:4049 , piece_id=5ICAF24EEJMQ7F75JYZCOHQ5GYTWOV2VYFROBIFAMS4SGPYM2P5A, seg_pos={0 0}, upload=536446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 03:37:20.325356005 (612h8m20.19643905s) [noise=true , address=145.79.227.40:4049 , piece_id=5ICAF24EEJMQ7F75JYZCOHQ5GYTWOV2VYFROBIFAMS4SGPYM2P5A, seg_pos={0 0}, upload=536446]: Long tail cancellation due to optimal threshold
Dec 31 03:37:20.325648362 (612h8m20.196731407s) [noise=true , address=145.79.227.37:4567 , piece_id=FHC26WAVVR5NJVMQCBMD5GLX26G2LVOCEGCQRRRHB3NA4HTMPQBA, seg_pos={0 0}, upload=536446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:37266->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:37266->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37266->145.79.227.37:4567: use of closed network connection
Dec 31 03:37:20.325664994 (612h8m20.196748029s) [noise=true , address=145.79.227.37:4567 , piece_id=FHC26WAVVR5NJVMQCBMD5GLX26G2LVOCEGCQRRRHB3NA4HTMPQBA, seg_pos={0 0}, upload=536446]: Long tail cancellation due to optimal threshold
Dec 31 03:37:20.325721371 (612h8m20.196804406s) [seg_pos={0 0}, upload=536446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:37:20.327143719 (612h8m20.198226754s) [upload=536446]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:37:20.327155431 (612h8m20.198238466s) [upload=536446]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:37:20.616968171 (612h8m20.488051206s) [upload=536446]: Done uploading segments...
Dec 31 03:37:20.616994711 (612h8m20.488077746s) [upload=536446]: Upload finished. err: <nil>
Duration: 30.003308681s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
56.523622761s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:37:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 56.52365878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................................X....X.X..X.......] duration:
17.323433812s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:37:08 UTC)
orbiter warning: context deadline exceeded
Duration: 17.323449162s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.......X..........X.......X.X...X...X...X.X.X...X......X..XX] duration:
5.001259804s trace:
https://jaeger.dev.storj.io/trace/76c3afa779b535f1
FAIL (2025-12-31 03:36:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001281044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.XX.X...X.XX..XX..X.X.X.XX.XX...X.X.......X....X..X..X....X.] duration:
6.61687915s trace:
https://jaeger.dev.storj.io/trace/67385290bac0c6ed
FAIL (2025-12-31 03:36:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: GRFYML35Y2KXYBCXWXXNRWGXRAFZESBSJKX34SXLGGWKWSQK2LSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 2JTL5MVZ2X2NDUWIV26MZWEMZC56G36IZPQ236B4ILN26B2EOXRQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: QXY4OS4BJI7S32KMOHH54YRG6Z76TK63RVCOO6N3WYG2RIIW2G4A) context deadline exceeded
======LOGS======
Dec 31 03:34:57.413173038 (612h5m57.284256074s) [upload=536410]: Starting upload
Dec 31 03:34:57.413255395 (612h5m57.28433845s) [upload=536410]: Done starting upload
Dec 31 03:34:57.413289500 (612h5m57.284372535s) [upload=536410]: Uploading segments...
Dec 31 03:34:57.413477598 (612h5m57.284560633s) [upload=536410]: Got next segment. Inline: false
Dec 31 03:34:57.413513746 (612h5m57.284596792s) [upload=536410]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:34:57.413518997 (612h5m57.284602042s) [upload=536410]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:34:57.661973033 (612h5m57.533056078s) [seg_pos={0 0}, upload=536410]: Begin upload segment...
Dec 31 03:34:57.662035643 (612h5m57.533118678s) [seg_pos={0 0}, upload=536410]: Done begin upload segment.
Dec 31 03:34:57.662080298 (612h5m57.533163333s) [noise=true , address=145.79.227.40:4051 , piece_id=4CPOVFDZ6DIWU5TUUXEJVP36CVTR6WBISDU5ZJ2UV7TL6YTFRB7Q, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.662153137 (612h5m57.533236172s) [noise=true , address=145.79.227.13:4802 , piece_id=3SJEBNTW4DAPJHFI2PUQVZN3XIART4IYLTL7GOMLQJWO4PTBHJVQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.662811139 (612h5m57.533894174s) [noise=true , address=45.91.116.6:4976 , piece_id=QXY4OS4BJI7S32KMOHH54YRG6Z76TK63RVCOO6N3WYG2RIIW2G4A, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.663143933 (612h5m57.534226978s) [noise=true , address=145.79.227.42:4123 , piece_id=OWRNTS4T46KK3S3P6EOUKJ7JSCK4VQ7RFAKGSMFL67TDHMY3ODZA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.663506493 (612h5m57.534589528s) [noise=true , address=145.79.227.8:4329 , piece_id=R6JVSPVDPFAMZCG5XECOJNMIRYEUHO557CEQAEF6DOWUKJ3UPNZA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.663773943 (612h5m57.534856978s) [noise=true , address=145.79.227.36:4517 , piece_id=YKIMTHDOPEZFPTQU2BW2MB2Q7JBIIBXJSDLQPVXD3IRAIRMRSOBA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.663910894 (612h5m57.534993939s) [noise=true , address=77.90.31.12:4590 , piece_id=XPMLE4AVITE4QVUIDKJA45ZSEMQJUFP54AQLKYJYDTQQ5FXYHWNA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.664030361 (612h5m57.535113396s) [noise=true , address=145.79.227.52:4445 , piece_id=CNIQ6ILGLXOJF4MFQURV72GR5XPFVQ3G3NKH5NNI2Q2IJW4SGZVA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.664330483 (612h5m57.535413518s) [noise=true , address=45.91.116.10:4081 , piece_id=QE7R6TY4PPOJWQLPU3ZJLXDBG6Y34GRMIWTZVORHJXZJXODZMNAQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.664357003 (612h5m57.535440048s) [noise=true , address=145.79.227.10:4403 , piece_id=L3LS336CHXL4C2UNB4UIY62YHSSOIXXL7HG6ZILS7V623G5HJELQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.664729392 (612h5m57.535812427s) [noise=true , address=45.91.116.12:4226 , piece_id=GRFYML35Y2KXYBCXWXXNRWGXRAFZESBSJKX34SXLGGWKWSQK2LSA, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.664829833 (612h5m57.535912868s) [noise=true , address=145.79.227.31:4799 , piece_id=RL753KMTOSGBONTIMGWW4PON6GROKF5FQTUVJGYGXG5FPLEFVOFQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.665047948 (612h5m57.536130983s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRHZEO4OFDCWAYKZ65UUGWTWDDNB7W4NPIFXTQ4OX73QAXPXAR2A, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.665303646 (612h5m57.536386691s) [noise=true , address=145.79.227.4:4187 , piece_id=5T7HOVJESIHWDJD6JO7RW7ZHXV6622UDDZUL43V2LHMCQJVCQCVQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.666283622 (612h5m57.537366667s) [noise=true , address=145.79.227.3:4150 , piece_id=7I72PFHHRZZAVENLCHBCZSGMKU5PNTGRLZB7QYGARCXUYXYFFC6A, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:57.667084507 (612h5m57.538167542s) [noise=true , address=77.90.31.10:4518 , piece_id=2JTL5MVZ2X2NDUWIV26MZWEMZC56G36IZPQ236B4ILN26B2EOXRQ, seg_pos={0 0}, upload=536410]: Uploading piece...
Dec 31 03:34:59.267730412 (612h5m59.138813447s) [upload=536410]: Next returned nil segment
Dec 31 03:34:59.267746652 (612h5m59.138829697s) [upload=536410]: Waiting for error group managing segments...
Dec 31 03:34:59.647847120 (612h5m59.518930185s) [noise=true , address=145.79.227.10:4403 , piece_id=L3LS336CHXL4C2UNB4UIY62YHSSOIXXL7HG6ZILS7V623G5HJELQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.650679583 (612h5m59.521762638s) [noise=true , address=145.79.227.13:4802 , piece_id=3SJEBNTW4DAPJHFI2PUQVZN3XIART4IYLTL7GOMLQJWO4PTBHJVQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.650935690 (612h5m59.522018725s) [noise=true , address=145.79.227.3:4150 , piece_id=7I72PFHHRZZAVENLCHBCZSGMKU5PNTGRLZB7QYGARCXUYXYFFC6A, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.665106284 (612h5m59.536189339s) [noise=true , address=145.79.227.4:4187 , piece_id=5T7HOVJESIHWDJD6JO7RW7ZHXV6622UDDZUL43V2LHMCQJVCQCVQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.671497763 (612h5m59.542580809s) [noise=true , address=145.79.227.31:4799 , piece_id=RL753KMTOSGBONTIMGWW4PON6GROKF5FQTUVJGYGXG5FPLEFVOFQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.671943773 (612h5m59.543026808s) [noise=true , address=145.79.227.36:4517 , piece_id=YKIMTHDOPEZFPTQU2BW2MB2Q7JBIIBXJSDLQPVXD3IRAIRMRSOBA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.700325575 (612h5m59.571408611s) [noise=true , address=145.79.227.52:4445 , piece_id=CNIQ6ILGLXOJF4MFQURV72GR5XPFVQ3G3NKH5NNI2Q2IJW4SGZVA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.801498190 (612h5m59.672581225s) [noise=true , address=145.79.227.8:4329 , piece_id=R6JVSPVDPFAMZCG5XECOJNMIRYEUHO557CEQAEF6DOWUKJ3UPNZA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.810175191 (612h5m59.681258227s) [noise=true , address=145.79.227.42:4123 , piece_id=OWRNTS4T46KK3S3P6EOUKJ7JSCK4VQ7RFAKGSMFL67TDHMY3ODZA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.908155518 (612h5m59.779238553s) [noise=true , address=77.90.31.10:4518 , piece_id=2JTL5MVZ2X2NDUWIV26MZWEMZC56G36IZPQ236B4ILN26B2EOXRQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:34:59.951338855 (612h5m59.82242189s) [noise=true , address=45.91.116.12:4226 , piece_id=GRFYML35Y2KXYBCXWXXNRWGXRAFZESBSJKX34SXLGGWKWSQK2LSA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:35:00.042205987 (612h5m59.913289022s) [noise=true , address=45.91.116.10:4081 , piece_id=QE7R6TY4PPOJWQLPU3ZJLXDBG6Y34GRMIWTZVORHJXZJXODZMNAQ, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:35:00.310530839 (612h6m0.181613875s) [noise=true , address=45.91.116.6:4976 , piece_id=QXY4OS4BJI7S32KMOHH54YRG6Z76TK63RVCOO6N3WYG2RIIW2G4A, seg_pos={0 0}, upload=536410]: Done uploading piece. err: <nil>
Dec 31 03:35:00.310698539 (612h6m0.181781584s) [seg_pos={0 0}, upload=536410]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:35:00.310971799 (612h6m0.182054835s) [noise=true , address=145.79.227.40:4051 , piece_id=4CPOVFDZ6DIWU5TUUXEJVP36CVTR6WBISDU5ZJ2UV7TL6YTFRB7Q, seg_pos={0 0}, upload=536410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 31 03:35:00.310982680 (612h6m0.182065715s) [noise=true , address=145.79.227.40:4051 , piece_id=4CPOVFDZ6DIWU5TUUXEJVP36CVTR6WBISDU5ZJ2UV7TL6YTFRB7Q, seg_pos={0 0}, upload=536410]: Long tail cancellation due to optimal threshold
Dec 31 03:35:00.311001506 (612h6m0.182084551s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRHZEO4OFDCWAYKZ65UUGWTWDDNB7W4NPIFXTQ4OX73QAXPXAR2A, seg_pos={0 0}, upload=536410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 31 03:35:00.311018318 (612h6m0.182101363s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRHZEO4OFDCWAYKZ65UUGWTWDDNB7W4NPIFXTQ4OX73QAXPXAR2A, seg_pos={0 0}, upload=536410]: Long tail cancellation due to optimal threshold
Dec 31 03:35:00.311193782 (612h6m0.182276817s) [noise=true , address=77.90.31.12:4590 , piece_id=XPMLE4AVITE4QVUIDKJA45ZSEMQJUFP54AQLKYJYDTQQ5FXYHWNA, seg_pos={0 0}, upload=536410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 31 03:35:00.311201186 (612h6m0.182284221s) [noise=true , address=77.90.31.12:4590 , piece_id=XPMLE4AVITE4QVUIDKJA45ZSEMQJUFP54AQLKYJYDTQQ5FXYHWNA, seg_pos={0 0}, upload=536410]: Long tail cancellation due to optimal threshold
Dec 31 03:35:00.311224801 (612h6m0.182307836s) [seg_pos={0 0}, upload=536410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:35:00.311311385 (612h6m0.18239442s) [upload=536410]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:35:00.311317107 (612h6m0.182400152s) [upload=536410]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:35:00.600318290 (612h6m0.471401335s) [upload=536410]: Done uploading segments...
Dec 31 03:35:00.600352616 (612h6m0.471435661s) [upload=536410]: Upload finished. err: <nil>
Duration: 6.616912033s
core
uplink
us-select-1
|