Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................X.........X...X.......] duration:
8.779570556s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:36:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.779597177s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.] duration:
4.737500357s trace:
https://jaeger.dev.storj.io/trace/544cf69ea397a218
FAIL (2025-12-08 15:36:59 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.737559008s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....................X............X.X.XX.XX.XX..X.....XX...X.] duration:
23.735458373s trace:
https://jaeger.dev.storj.io/trace/4137901a093aef85
FAIL (2025-12-08 15:36:47 UTC)
orbiter warning: context deadline exceeded
Duration: 23.735477679s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
18.891744243s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:36:37 UTC)
orbiter warning: context deadline exceeded
Duration: 18.891783989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X] duration:
5.001116763s trace:
https://jaeger.dev.storj.io/trace/544cf69ea397a218
FAIL (2025-12-08 15:36:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001138835s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X.........X...X......] duration:
12.318443196s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:36:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 12.318478431s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
8.104661477s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:36:25 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.104687506s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
10.344237854s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:36:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.34425593s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
18.964928266s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:36:08 UTC)
orbiter warning: context deadline exceeded
Duration: 18.964941713s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....................X............X.X.XX.XX.XX..X.....XX...X] duration:
30.001408973s trace:
https://jaeger.dev.storj.io/trace/4137901a093aef85
FAIL (2025-12-08 15:36:00 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001426967s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
25.646888397s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:36:00 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 25.646916531s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
20.994398126s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:35:46 UTC)
orbiter warning: context deadline exceeded
Duration: 20.99443196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.] duration:
3.556066659s trace:
https://jaeger.dev.storj.io/trace/6285853196ba3393
FAIL (2025-12-08 15:35:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.556091075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
7.61430217s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:35:32 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.614329381s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [......................X............X.X.XX.XX.XX..X.....XX...] duration:
23.401896733s trace:
https://jaeger.dev.storj.io/trace/63b11df93e911a70
FAIL (2025-12-08 15:35:30 UTC)
orbiter warning: context deadline exceeded
Duration: 23.401914778s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................................X.........X...X.....] duration:
8.99838149s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:35:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.99841291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX] duration:
5.002213488s trace:
https://jaeger.dev.storj.io/trace/6285853196ba3393
FAIL (2025-12-08 15:35:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002236884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
19.503612414s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:35:24 UTC)
orbiter warning: context deadline exceeded
Duration: 19.503697115s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X.........X...X....] duration:
8.928032692s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:35:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.928065224s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.......................X............X.X.XX.XX.XX..X.....XX..] duration:
19.736257029s trace:
https://jaeger.dev.storj.io/trace/63b11df93e911a70
FAIL (2025-12-08 15:35:06 UTC)
orbiter warning: context deadline exceeded
Duration: 19.736285364s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
8.929201532s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:34:50 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.929231608s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [........................X............X.X.XX.XX.XX..X.....XX.] duration:
20.136212218s trace:
https://jaeger.dev.storj.io/trace/63b11df93e911a70
FAIL (2025-12-08 15:34:46 UTC)
orbiter warning: context deadline exceeded
Duration: 20.136244961s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
7.698559229s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:34:29 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.698590047s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.........................X............X.X.XX.XX.XX..X.....XX] duration:
30.001014139s trace:
https://jaeger.dev.storj.io/trace/63b11df93e911a70
FAIL (2025-12-08 15:34:19 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001041181s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....X] duration:
5.001035536s trace:
https://jaeger.dev.storj.io/trace/7bf63ff941bb8ae4
FAIL (2025-12-08 15:34:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00106423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
9.027348687s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:34:12 UTC)
orbiter warning: context deadline exceeded
Duration: 9.027379818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X.........X...X...] duration:
8.478967304s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:34:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.478999555s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
17.810987504s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:34:01 UTC)
orbiter warning: context deadline exceeded
Duration: 17.811023702s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
10.178481567s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:33:49 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.178509509s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
7.408025365s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:33:38 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.408054692s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
15.010963684s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:33:31 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 15.010985144s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....] duration:
3.51640216s trace:
https://jaeger.dev.storj.io/trace/567cf271cf4a7926
FAIL (2025-12-08 15:33:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.516493553s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..........................X............X.X.XX.XX.XX..X.....X] duration:
30.000930066s trace:
https://jaeger.dev.storj.io/trace/65ef97fa320b0f6f
FAIL (2025-12-08 15:33:19 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000954673s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
12.58410305s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:33:16 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 12.584130341s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [X.........................................X.................] duration:
6.627550087s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:33:12 UTC)
orbiter warning: context deadline exceeded
Duration: 6.627639106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
8.40774531s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:33:10 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.407779267s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
13.865222832s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:33:10 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 13.865254522s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X.........X...X..] duration:
8.727996455s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:33:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.72802584s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
11.246303957s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:32:51 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 11.246353881s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................................X.........X...X.] duration:
9.342819812s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:32:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 9.343892636s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
13.597360727s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:32:38 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 13.597379111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X....] duration:
4.445972302s trace:
https://jaeger.dev.storj.io/trace/567cf271cf4a7926
FAIL (2025-12-08 15:32:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.446005665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
8.122787848s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:32:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 8.122817955s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
30.456294218s trace:
https://jaeger.dev.storj.io/trace/29daaac43eca6484
FAIL (2025-12-08 15:32:24 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 30.456325668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.X.........................................X................] duration:
6.176842503s trace:
https://jaeger.dev.storj.io/trace/5cf02331713757d4
FAIL (2025-12-08 15:32:20 UTC)
orbiter warning: context deadline exceeded
Duration: 6.176917826s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X.........X...X] duration:
30.003110465s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:32:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 30.003137767s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
6.526255216s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:32:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.526290122s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [...........................X............X.X.XX.XX.XX..X.....] duration:
16.733637575s trace:
https://jaeger.dev.storj.io/trace/397564ec35512283
FAIL (2025-12-08 15:31:58 UTC)
orbiter warning: context deadline exceeded
Duration: 16.733659607s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
18.620667006s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:31:58 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 18.620686351s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X...] duration:
3.57132257s trace:
https://jaeger.dev.storj.io/trace/567cf271cf4a7926
FAIL (2025-12-08 15:31:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.571351655s
core
uplink
us-select-1
us1-placement-20
|