Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [.X.....XX....X...........X....................X............X] duration:
30.001806431s trace:
https://jaeger.dev.storj.io/trace/303958fb6bfcc95e
FAIL (2025-12-28 13:39:14 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001828395s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X..........X.XX..XX..X.......XXX......XX......X..X.....X..X.] duration:
6.717108028s trace:
https://jaeger.dev.storj.io/trace/6c1591a05ad2eac6
FAIL (2025-12-28 13:38:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:38:08.464226011 (550h9m8.335309046s) [upload=482354]: Starting upload
Dec 28 13:38:08.464290404 (550h9m8.335373449s) [upload=482354]: Done starting upload
Dec 28 13:38:08.464378000 (550h9m8.335461045s) [upload=482354]: Uploading segments...
Dec 28 13:38:08.464567011 (550h9m8.335650056s) [upload=482354]: Got next segment. Inline: false
Dec 28 13:38:08.464622396 (550h9m8.335705441s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:38:08.464627546 (550h9m8.335710581s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:38:08.711212654 (550h9m8.582295689s) [seg_pos={0 0}, upload=482354]: Begin upload segment...
Dec 28 13:38:08.711289841 (550h9m8.582372876s) [seg_pos={0 0}, upload=482354]: Done begin upload segment.
Dec 28 13:38:08.711362389 (550h9m8.582445434s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.711400963 (550h9m8.582484008s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.712394153 (550h9m8.583477198s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.712489735 (550h9m8.583572781s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713144693 (550h9m8.584227738s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713619085 (550h9m8.584702131s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713652098 (550h9m8.584735133s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714338365 (550h9m8.585421411s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714603731 (550h9m8.585686776s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714665359 (550h9m8.585748394s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.715357636 (550h9m8.586440681s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.715714086 (550h9m8.586797121s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.717021004 (550h9m8.588104049s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.719023978 (550h9m8.590107013s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.723965698 (550h9m8.595048743s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.732674431 (550h9m8.603757496s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:10.431589151 (550h9m10.302672196s) [upload=482354]: Next returned nil segment
Dec 28 13:38:10.431644125 (550h9m10.302727161s) [upload=482354]: Waiting for error group managing segments...
Dec 28 13:38:10.727468924 (550h9m10.598551969s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.728722090 (550h9m10.599805135s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.737014090 (550h9m10.608097135s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.741090482 (550h9m10.612173527s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.745833063 (550h9m10.616916108s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.749513942 (550h9m10.620596987s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.749569217 (550h9m10.620652252s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.754150741 (550h9m10.625233776s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.765496985 (550h9m10.636580031s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.770863013 (550h9m10.641946058s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.771430154 (550h9m10.642513199s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.951904947 (550h9m10.822987993s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:11.007731691 (550h9m10.878814736s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:11.007758952 (550h9m10.878841997s) [seg_pos={0 0}, upload=482354]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:38:11.008042753 (550h9m10.879125788s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 28 13:38:11.008050788 (550h9m10.879133833s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008219600 (550h9m10.879302645s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:38:11.008226793 (550h9m10.879309828s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008453564 (550h9m10.879536609s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43300->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:38:11.008461540 (550h9m10.879544585s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008487138 (550h9m10.879570183s) [seg_pos={0 0}, upload=482354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:38:11.008569305 (550h9m10.87965235s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:38:11.008574235 (550h9m10.87965728s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:38:11.296594462 (550h9m11.167677507s) [upload=482354]: Done uploading segments...
Dec 28 13:38:11.296702798 (550h9m11.167785833s) [upload=482354]: Upload finished. err: <nil>
Duration: 6.717141482s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X..........X.XX..XX..X.......XXX......XX......X..X.....X..X] duration:
30.002879161s trace:
https://jaeger.dev.storj.io/trace/6c1591a05ad2eac6
FAIL (2025-12-28 13:38:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:38:08.464226011 (550h9m8.335309046s) [upload=482354]: Starting upload
Dec 28 13:38:08.464290404 (550h9m8.335373449s) [upload=482354]: Done starting upload
Dec 28 13:38:08.464378000 (550h9m8.335461045s) [upload=482354]: Uploading segments...
Dec 28 13:38:08.464567011 (550h9m8.335650056s) [upload=482354]: Got next segment. Inline: false
Dec 28 13:38:08.464622396 (550h9m8.335705441s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:38:08.464627546 (550h9m8.335710581s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:38:08.711212654 (550h9m8.582295689s) [seg_pos={0 0}, upload=482354]: Begin upload segment...
Dec 28 13:38:08.711289841 (550h9m8.582372876s) [seg_pos={0 0}, upload=482354]: Done begin upload segment.
Dec 28 13:38:08.711362389 (550h9m8.582445434s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.711400963 (550h9m8.582484008s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.712394153 (550h9m8.583477198s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.712489735 (550h9m8.583572781s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713144693 (550h9m8.584227738s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713619085 (550h9m8.584702131s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.713652098 (550h9m8.584735133s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714338365 (550h9m8.585421411s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714603731 (550h9m8.585686776s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.714665359 (550h9m8.585748394s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.715357636 (550h9m8.586440681s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.715714086 (550h9m8.586797121s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.717021004 (550h9m8.588104049s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.719023978 (550h9m8.590107013s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.723965698 (550h9m8.595048743s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:08.732674431 (550h9m8.603757496s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Uploading piece...
Dec 28 13:38:10.431589151 (550h9m10.302672196s) [upload=482354]: Next returned nil segment
Dec 28 13:38:10.431644125 (550h9m10.302727161s) [upload=482354]: Waiting for error group managing segments...
Dec 28 13:38:10.727468924 (550h9m10.598551969s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.728722090 (550h9m10.599805135s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.737014090 (550h9m10.608097135s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.741090482 (550h9m10.612173527s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.745833063 (550h9m10.616916108s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.749513942 (550h9m10.620596987s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.749569217 (550h9m10.620652252s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.754150741 (550h9m10.625233776s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.765496985 (550h9m10.636580031s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.770863013 (550h9m10.641946058s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.771430154 (550h9m10.642513199s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:10.951904947 (550h9m10.822987993s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:11.007731691 (550h9m10.878814736s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil>
Dec 28 13:38:11.007758952 (550h9m10.878841997s) [seg_pos={0 0}, upload=482354]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:38:11.008042753 (550h9m10.879125788s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 28 13:38:11.008050788 (550h9m10.879133833s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008219600 (550h9m10.879302645s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 28 13:38:11.008226793 (550h9m10.879309828s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008453564 (550h9m10.879536609s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43300->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:38:11.008461540 (550h9m10.879544585s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold
Dec 28 13:38:11.008487138 (550h9m10.879570183s) [seg_pos={0 0}, upload=482354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:38:11.008569305 (550h9m10.87965235s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:38:11.008574235 (550h9m10.87965728s) [upload=482354]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:38:11.296594462 (550h9m11.167677507s) [upload=482354]: Done uploading segments...
Dec 28 13:38:11.296702798 (550h9m11.167785833s) [upload=482354]: Upload finished. err: <nil>
Duration: 30.002903997s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X.....XX....X...........X....................X............] duration:
15.641888775s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:38:26 UTC)
orbiter warning: context deadline exceeded
Duration: 15.642009335s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXX] duration:
5.001581004s trace:
https://jaeger.dev.storj.io/trace/6d2059f6d946a68e
FAIL (2025-12-28 13:38:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001605871s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
7.756527322s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 13:38:12 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.756544755s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X.....XX....X...........X....................X...........] duration:
21.563481733s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:38:09 UTC)
orbiter warning: context deadline exceeded
Duration: 21.56358106s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.004703069s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 13:38:04 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 1m0.004732175s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X........X...............] duration:
32.21776502s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:38:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 32.217781191s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [.X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XX] duration:
5.000824984s trace:
https://jaeger.dev.storj.io/trace/4f42a5dcf68d2b65
FAIL (2025-12-28 13:37:47 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000840863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X.X..........X.XX..XX..X.......XXX......XX......X..X.....X..] duration:
5.97884784s trace:
https://jaeger.dev.storj.io/trace/784a3f13b4812d01
FAIL (2025-12-28 13:37:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA) context deadline exceeded
======LOGS======
Dec 28 13:35:50.062665028 (550h6m49.933748073s) [upload=482321]: Starting upload
Dec 28 13:35:50.062748207 (550h6m49.933831242s) [upload=482321]: Done starting upload
Dec 28 13:35:50.062809193 (550h6m49.933892228s) [upload=482321]: Uploading segments...
Dec 28 13:35:50.062965481 (550h6m49.934048516s) [upload=482321]: Got next segment. Inline: false
Dec 28 13:35:50.063021718 (550h6m49.934104753s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:35:50.063026357 (550h6m49.934109392s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:35:50.310224221 (550h6m50.181307286s) [seg_pos={0 0}, upload=482321]: Begin upload segment...
Dec 28 13:35:50.310295397 (550h6m50.181378432s) [seg_pos={0 0}, upload=482321]: Done begin upload segment.
Dec 28 13:35:50.310331626 (550h6m50.181414671s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310358276 (550h6m50.181441321s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310419774 (550h6m50.181502829s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310947418 (550h6m50.182030453s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311143151 (550h6m50.182226196s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311277658 (550h6m50.182360693s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311831001 (550h6m50.182914046s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312382591 (550h6m50.183465636s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312969849 (550h6m50.184052884s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313353800 (550h6m50.184436835s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313726510 (550h6m50.184809555s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313835117 (550h6m50.184918162s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314393881 (550h6m50.185476936s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314780888 (550h6m50.185863933s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315126686 (550h6m50.186209731s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315308213 (550h6m50.186391258s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:52.019101362 (550h6m51.890184408s) [upload=482321]: Next returned nil segment
Dec 28 13:35:52.019126300 (550h6m51.890209335s) [upload=482321]: Waiting for error group managing segments...
Dec 28 13:35:52.306862006 (550h6m52.177945041s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.312294029 (550h6m52.183377065s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.316843582 (550h6m52.187926617s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.320101154 (550h6m52.191184199s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.321281732 (550h6m52.192364767s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.325875530 (550h6m52.196958575s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.332572743 (550h6m52.203655788s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.336657431 (550h6m52.207740476s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.349638088 (550h6m52.220721133s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.585376954 (550h6m52.456459999s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.613580307 (550h6m52.484663352s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.634525711 (550h6m52.505608746s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679552599 (550h6m52.550635644s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679570724 (550h6m52.550653759s) [seg_pos={0 0}, upload=482321]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:35:52.679790232 (550h6m52.550873277s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.679800371 (550h6m52.550883416s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680032393 (550h6m52.551115428s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.680073130 (550h6m52.551156165s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680112565 (550h6m52.5511956s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection
Dec 28 13:35:52.680121052 (550h6m52.551204087s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680140990 (550h6m52.551224025s) [seg_pos={0 0}, upload=482321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:35:52.680209109 (550h6m52.551292164s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:35:52.680214860 (550h6m52.551297905s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:35:52.975175162 (550h6m52.846258197s) [upload=482321]: Done uploading segments...
Dec 28 13:35:52.975223415 (550h6m52.84630645s) [upload=482321]: Upload finished. err: <nil>
Duration: 5.978873007s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X........X..............] duration:
42.566972539s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:37:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 42.566986526s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.X] duration:
5.001503827s trace:
https://jaeger.dev.storj.io/trace/3b08ae91ad1a425f
FAIL (2025-12-28 13:37:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00152665s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X.....XX....X...........X....................X..........] duration:
15.523230526s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:37:04 UTC)
orbiter warning: context deadline exceeded
Duration: 15.523258308s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [X..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.] duration:
3.445978824s trace:
https://jaeger.dev.storj.io/trace/ec24379fcc82955
FAIL (2025-12-28 13:37:01 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.446008783s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX.X..........X.XX..XX..X.......XXX......XX......X..X.....X.] duration:
6.32293703s trace:
https://jaeger.dev.storj.io/trace/784a3f13b4812d01
FAIL (2025-12-28 13:36:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA) context deadline exceeded
======LOGS======
Dec 28 13:35:50.062665028 (550h6m49.933748073s) [upload=482321]: Starting upload
Dec 28 13:35:50.062748207 (550h6m49.933831242s) [upload=482321]: Done starting upload
Dec 28 13:35:50.062809193 (550h6m49.933892228s) [upload=482321]: Uploading segments...
Dec 28 13:35:50.062965481 (550h6m49.934048516s) [upload=482321]: Got next segment. Inline: false
Dec 28 13:35:50.063021718 (550h6m49.934104753s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:35:50.063026357 (550h6m49.934109392s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:35:50.310224221 (550h6m50.181307286s) [seg_pos={0 0}, upload=482321]: Begin upload segment...
Dec 28 13:35:50.310295397 (550h6m50.181378432s) [seg_pos={0 0}, upload=482321]: Done begin upload segment.
Dec 28 13:35:50.310331626 (550h6m50.181414671s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310358276 (550h6m50.181441321s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310419774 (550h6m50.181502829s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310947418 (550h6m50.182030453s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311143151 (550h6m50.182226196s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311277658 (550h6m50.182360693s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311831001 (550h6m50.182914046s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312382591 (550h6m50.183465636s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312969849 (550h6m50.184052884s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313353800 (550h6m50.184436835s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313726510 (550h6m50.184809555s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313835117 (550h6m50.184918162s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314393881 (550h6m50.185476936s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314780888 (550h6m50.185863933s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315126686 (550h6m50.186209731s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315308213 (550h6m50.186391258s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:52.019101362 (550h6m51.890184408s) [upload=482321]: Next returned nil segment
Dec 28 13:35:52.019126300 (550h6m51.890209335s) [upload=482321]: Waiting for error group managing segments...
Dec 28 13:35:52.306862006 (550h6m52.177945041s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.312294029 (550h6m52.183377065s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.316843582 (550h6m52.187926617s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.320101154 (550h6m52.191184199s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.321281732 (550h6m52.192364767s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.325875530 (550h6m52.196958575s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.332572743 (550h6m52.203655788s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.336657431 (550h6m52.207740476s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.349638088 (550h6m52.220721133s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.585376954 (550h6m52.456459999s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.613580307 (550h6m52.484663352s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.634525711 (550h6m52.505608746s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679552599 (550h6m52.550635644s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679570724 (550h6m52.550653759s) [seg_pos={0 0}, upload=482321]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:35:52.679790232 (550h6m52.550873277s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.679800371 (550h6m52.550883416s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680032393 (550h6m52.551115428s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.680073130 (550h6m52.551156165s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680112565 (550h6m52.5511956s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection
Dec 28 13:35:52.680121052 (550h6m52.551204087s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680140990 (550h6m52.551224025s) [seg_pos={0 0}, upload=482321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:35:52.680209109 (550h6m52.551292164s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:35:52.680214860 (550h6m52.551297905s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:35:52.975175162 (550h6m52.846258197s) [upload=482321]: Done uploading segments...
Dec 28 13:35:52.975223415 (550h6m52.84630645s) [upload=482321]: Upload finished. err: <nil>
Duration: 6.322967128s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X........X.............] duration:
45.595509257s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:36:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 45.595538653s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX] duration:
5.001820839s trace:
https://jaeger.dev.storj.io/trace/ec24379fcc82955
FAIL (2025-12-28 13:36:36 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00184745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X.....XX....X...........X....................X.........] duration:
18.535609036s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:36:27 UTC)
orbiter warning: context deadline exceeded
Duration: 18.535632481s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX.X..........X.XX..XX..X.......XXX......XX......X..X.....X] duration:
30.003313585s trace:
https://jaeger.dev.storj.io/trace/784a3f13b4812d01
FAIL (2025-12-28 13:36:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA) context deadline exceeded
======LOGS======
Dec 28 13:35:50.062665028 (550h6m49.933748073s) [upload=482321]: Starting upload
Dec 28 13:35:50.062748207 (550h6m49.933831242s) [upload=482321]: Done starting upload
Dec 28 13:35:50.062809193 (550h6m49.933892228s) [upload=482321]: Uploading segments...
Dec 28 13:35:50.062965481 (550h6m49.934048516s) [upload=482321]: Got next segment. Inline: false
Dec 28 13:35:50.063021718 (550h6m49.934104753s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:35:50.063026357 (550h6m49.934109392s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:35:50.310224221 (550h6m50.181307286s) [seg_pos={0 0}, upload=482321]: Begin upload segment...
Dec 28 13:35:50.310295397 (550h6m50.181378432s) [seg_pos={0 0}, upload=482321]: Done begin upload segment.
Dec 28 13:35:50.310331626 (550h6m50.181414671s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310358276 (550h6m50.181441321s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310419774 (550h6m50.181502829s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.310947418 (550h6m50.182030453s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311143151 (550h6m50.182226196s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311277658 (550h6m50.182360693s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.311831001 (550h6m50.182914046s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312382591 (550h6m50.183465636s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.312969849 (550h6m50.184052884s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313353800 (550h6m50.184436835s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313726510 (550h6m50.184809555s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.313835117 (550h6m50.184918162s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314393881 (550h6m50.185476936s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.314780888 (550h6m50.185863933s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315126686 (550h6m50.186209731s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:50.315308213 (550h6m50.186391258s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Uploading piece...
Dec 28 13:35:52.019101362 (550h6m51.890184408s) [upload=482321]: Next returned nil segment
Dec 28 13:35:52.019126300 (550h6m51.890209335s) [upload=482321]: Waiting for error group managing segments...
Dec 28 13:35:52.306862006 (550h6m52.177945041s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.312294029 (550h6m52.183377065s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.316843582 (550h6m52.187926617s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.320101154 (550h6m52.191184199s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.321281732 (550h6m52.192364767s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.325875530 (550h6m52.196958575s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.332572743 (550h6m52.203655788s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.336657431 (550h6m52.207740476s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.349638088 (550h6m52.220721133s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.585376954 (550h6m52.456459999s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.613580307 (550h6m52.484663352s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.634525711 (550h6m52.505608746s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679552599 (550h6m52.550635644s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil>
Dec 28 13:35:52.679570724 (550h6m52.550653759s) [seg_pos={0 0}, upload=482321]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:35:52.679790232 (550h6m52.550873277s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.679800371 (550h6m52.550883416s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680032393 (550h6m52.551115428s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF
Dec 28 13:35:52.680073130 (550h6m52.551156165s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680112565 (550h6m52.5511956s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection
Dec 28 13:35:52.680121052 (550h6m52.551204087s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold
Dec 28 13:35:52.680140990 (550h6m52.551224025s) [seg_pos={0 0}, upload=482321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:35:52.680209109 (550h6m52.551292164s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:35:52.680214860 (550h6m52.551297905s) [upload=482321]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:35:52.975175162 (550h6m52.846258197s) [upload=482321]: Done uploading segments...
Dec 28 13:35:52.975223415 (550h6m52.84630645s) [upload=482321]: Upload finished. err: <nil>
Duration: 30.003335275s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXX] duration:
5.001907147s trace:
https://jaeger.dev.storj.io/trace/47eb062b4d9c0d64
FAIL (2025-12-28 13:36:04 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001978593s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X........X............] duration:
47.350306031s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:36:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 47.350336018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X.....XX....X...........X....................X........] duration:
19.870546797s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:35:46 UTC)
orbiter warning: context deadline exceeded
Duration: 19.870565482s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX.X..........X.XX..XX..X.......XXX......XX......X..X.....] duration:
5.948998533s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:35:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 5.949026487s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X........X...........] duration:
44.576983902s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:35:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 44.5770095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...XX.X..........X.XX..XX..X.......XXX......XX......X..X....] duration:
6.052762437s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:35:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 6.052793156s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXX] duration:
5.001438092s trace:
https://jaeger.dev.storj.io/trace/5d812b8f0e01cb5c
FAIL (2025-12-28 13:34:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001464562s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X.....XX....X...........X....................X.......] duration:
19.328525475s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:34:50 UTC)
orbiter warning: context deadline exceeded
Duration: 19.328542229s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X.....XX....X...........X....................X......] duration:
19.453988328s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:34:31 UTC)
orbiter warning: context deadline exceeded
Duration: 19.454013137s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....XX.X..........X.XX..XX..X.......XXX......XX......X..X...] duration:
5.689789997s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:34:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 5.689822939s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X........X..........] duration:
28.116025646s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:34:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 28.116040424s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.....XX.X..........X.XX..XX..X.......XXX......XX......X..X..] duration:
5.686489972s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:33:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 5.686542063s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXX] duration:
5.001759574s trace:
https://jaeger.dev.storj.io/trace/53749821fdf9ec64
FAIL (2025-12-28 13:33:50 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001783973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........X.....XX....X...........X....................X.....] duration:
15.169613735s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:33:34 UTC)
orbiter warning: context deadline exceeded
Duration: 15.169644264s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X........X.........] duration:
58.609433588s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:33:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 58.609453227s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [......XX.X..........X.XX..XX..X.......XXX......XX......X..X.] duration:
5.755556066s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:33:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 5.755584189s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........X.....XX....X...........X....................X....] duration:
18.498413371s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:33:15 UTC)
orbiter warning: context deadline exceeded
Duration: 18.498484535s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXX] duration:
5.001472898s trace:
https://jaeger.dev.storj.io/trace/244386aadc6ed931
FAIL (2025-12-28 13:33:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001499849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.......XX.X..........X.XX..XX..X.......XXX......XX......X..X] duration:
30.004700711s trace:
https://jaeger.dev.storj.io/trace/4b9b6a438213a8d3
FAIL (2025-12-28 13:33:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ) context deadline exceeded; piecestore: (Node ID: 1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, Piece ID: 3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A) context deadline exceeded
======LOGS======
Dec 28 13:32:33.002358608 (550h3m32.873441663s) [upload=482276]: Starting upload
Dec 28 13:32:33.002447837 (550h3m32.873530902s) [upload=482276]: Done starting upload
Dec 28 13:32:33.002591150 (550h3m32.873674195s) [upload=482276]: Uploading segments...
Dec 28 13:32:33.002753169 (550h3m32.873836224s) [upload=482276]: Got next segment. Inline: false
Dec 28 13:32:33.002811530 (550h3m32.873894585s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:32:33.002819145 (550h3m32.8739022s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:32:33.247993006 (550h3m33.119076051s) [seg_pos={0 0}, upload=482276]: Begin upload segment...
Dec 28 13:32:33.248079580 (550h3m33.119162625s) [seg_pos={0 0}, upload=482276]: Done begin upload segment.
Dec 28 13:32:33.248121750 (550h3m33.119204796s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.248208536 (550h3m33.119291571s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249059396 (550h3m33.120142431s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249157222 (550h3m33.120240257s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249530353 (550h3m33.120613398s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.249658547 (550h3m33.120741582s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250120958 (550h3m33.121203993s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250148721 (550h3m33.121231766s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250439063 (550h3m33.121522098s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250634445 (550h3m33.12171749s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250714578 (550h3m33.121797623s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.250995353 (550h3m33.122078388s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251115211 (550h3m33.122198246s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251298089 (550h3m33.122381125s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.251560509 (550h3m33.122643554s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:33.252067014 (550h3m33.123150059s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Uploading piece...
Dec 28 13:32:34.958028252 (550h3m34.829111287s) [upload=482276]: Next returned nil segment
Dec 28 13:32:34.958047918 (550h3m34.829130994s) [upload=482276]: Waiting for error group managing segments...
Dec 28 13:32:35.250704184 (550h3m35.121787219s) [noise=true , address=145.79.227.20:4439 , piece_id=JA5YJUJPD7PO3JE3VIXHEYZCKITQEON3NKVHCDUKQMWCLE4C76KQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252256369 (550h3m35.123339405s) [noise=true , address=145.79.227.19:4079 , piece_id=WPLGQU534WFNY5OUZ4BJPIG6X5SE7QX4RKL3BPITAUHXQX2QFQ7A, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.252724131 (550h3m35.123807166s) [noise=true , address=145.79.227.12:4767 , piece_id=3UX2XZZ3YLVBPDLU6G3XJDKLMW37RASVVNLPF52SRAKUE5XIINHQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.254506144 (550h3m35.125589189s) [noise=true , address=145.79.227.52:4445 , piece_id=RRIER24K5SXQEIQLXNO6DKZWLFVSAOPLKGDVWKFHO445LY5FGFZQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.259835472 (550h3m35.130918517s) [noise=true , address=145.79.227.11:4731 , piece_id=WUXSBDPI7TWLVWEOELHSFUKE36ULXBHT2Y4EGJOYNEAHQYZRWIEA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.261372689 (550h3m35.132455724s) [noise=true , address=145.79.227.29:4942 , piece_id=GGNCAPAB5APOPV3FW7AECAOJDIQO5R7VPT2XDEUFP6CEZFSMGLGQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.262099603 (550h3m35.133182638s) [noise=true , address=145.79.227.51:4410 , piece_id=WV3F5XERZBDRQPCVY4ECPTUYASRZ2LQ6ACL5WUMURGAU55HBMESA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.266932445 (550h3m35.13801549s) [noise=true , address=145.79.227.47:4266 , piece_id=STSZ7ZGCIR64SYYWWZUL4VYTD6CGDHIJ44JUPGF3ZMXITSHR6N3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.270613434 (550h3m35.141696479s) [noise=true , address=145.79.227.48:4302 , piece_id=RVXCD4N6XDP562ZO74AIDQMXVNA32XP4EVGFTOKJ2Z2TKSKJY2DQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.281706528 (550h3m35.152789563s) [noise=true , address=145.79.227.17:4971 , piece_id=K7FR4WE7526ZUBWKOY52QYXAHSENNHIBYOVCY7Z2AI2V2PL5H5ZA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.286167181 (550h3m35.157250226s) [noise=true , address=145.79.227.25:4619 , piece_id=LTTORXJGQ2BUDYYV253EKKRB7PB7WAPXOSTTGP2WVA4TP2PUYTYQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.458695042 (550h3m35.329778087s) [noise=true , address=77.90.31.16:4736 , piece_id=VKQMQSER54PRTGQBETFRQ7QFY7FIQKGRZDHDMXIBZZLQQOAAWT3Q, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508970815 (550h3m35.38005385s) [noise=true , address=77.90.31.15:4697 , piece_id=MTAJZQD6A6OVHTIDXMKZEPEZFYRABMFQFDFM6L7RVB5IAU3QWDLQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: <nil>
Dec 28 13:32:35.508993508 (550h3m35.380076543s) [seg_pos={0 0}, upload=482276]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:32:35.509180494 (550h3m35.380263529s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509186706 (550h3m35.380269741s) [noise=true , address=45.91.116.9:4261 , piece_id=CTPUAABI6UUIQGTTYRN6Y6P6CCWZZOXGNKGXCDCF4I5VJOGKYBLA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509356739 (550h3m35.380439774s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509365637 (550h3m35.380448682s) [noise=true , address=145.79.227.49:4338 , piece_id=GBYG25UXMS7HTKUIFXLVBXSDIYBNM7QMUZTRL7KEZ6FK2Q46TQDA, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509395673 (550h3m35.380478718s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 28 13:32:35.509424098 (550h3m35.380507143s) [noise=true , address=45.91.116.11:4119 , piece_id=PCBJ2GGQV6DRW5R6WJNTNUF4XZOJU3X4J2J7IIEFHF5FZMY5C7JQ, seg_pos={0 0}, upload=482276]: Long tail cancellation due to optimal threshold
Dec 28 13:32:35.509450889 (550h3m35.380533934s) [seg_pos={0 0}, upload=482276]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:32:35.509530400 (550h3m35.380613435s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:32:35.509536793 (550h3m35.380619828s) [upload=482276]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:32:35.803906000 (550h3m35.674989045s) [upload=482276]: Done uploading segments...
Dec 28 13:32:35.803927249 (550h3m35.675010294s) [upload=482276]: Upload finished. err: <nil>
Duration: 30.00474711s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........X.....XX....X...........X....................X...] duration:
18.559892676s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:32:54 UTC)
orbiter warning: context deadline exceeded
Duration: 18.559964011s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X........X........] duration:
50.105826003s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:32:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 50.105851342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XX] duration:
5.001584737s trace:
https://jaeger.dev.storj.io/trace/58d25a44986072cd
FAIL (2025-12-28 13:32:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001615185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X.....XX....X...........X....................X..] duration:
19.085990493s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:31:42 UTC)
orbiter warning: context deadline exceeded
Duration: 19.086016613s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [..XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.X] duration:
5.001443561s trace:
https://jaeger.dev.storj.io/trace/62cf28d202af8ec4
FAIL (2025-12-28 13:31:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001466256s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [........XX.X..........X.XX..XX..X.......XXX......XX......X..] duration:
5.812919264s trace:
https://jaeger.dev.storj.io/trace/2b041200f1e30347
FAIL (2025-12-28 13:31:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:29:58.639531113 (550h0m58.510614158s) [upload=482242]: Starting upload
Dec 28 13:29:58.639669457 (550h0m58.510752492s) [upload=482242]: Done starting upload
Dec 28 13:29:58.639790927 (550h0m58.510873972s) [upload=482242]: Uploading segments...
Dec 28 13:29:58.640333441 (550h0m58.511416476s) [upload=482242]: Got next segment. Inline: false
Dec 28 13:29:58.640392624 (550h0m58.511475669s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:29:58.640414345 (550h0m58.51149739s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:29:58.886569264 (550h0m58.757652299s) [seg_pos={0 0}, upload=482242]: Begin upload segment...
Dec 28 13:29:58.886646741 (550h0m58.757729786s) [seg_pos={0 0}, upload=482242]: Done begin upload segment.
Dec 28 13:29:58.886672791 (550h0m58.757755836s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.887683495 (550h0m58.758766541s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888301993 (550h0m58.759385039s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888722093 (550h0m58.759805128s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889128457 (550h0m58.760211492s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889580348 (550h0m58.760663393s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889762865 (550h0m58.7608459s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891199751 (550h0m58.762282796s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891678813 (550h0m58.762761849s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892085699 (550h0m58.763168734s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892609716 (550h0m58.763692761s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893015538 (550h0m58.764098573s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893626081 (550h0m58.764709127s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894201036 (550h0m58.765284071s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894671662 (550h0m58.765754728s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.896608850 (550h0m58.767691886s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:30:00.517630309 (550h1m0.388713364s) [upload=482242]: Next returned nil segment
Dec 28 13:30:00.517644185 (550h1m0.38872723s) [upload=482242]: Waiting for error group managing segments...
Dec 28 13:30:00.902991830 (550h1m0.774074865s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.904178460 (550h1m0.775261505s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.913000588 (550h1m0.784083633s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.917507580 (550h1m0.788590615s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.919104200 (550h1m0.790187235s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.920957218 (550h1m0.792040253s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.921859647 (550h1m0.792942702s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.923841651 (550h1m0.794924706s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.925186592 (550h1m0.796269637s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.926103869 (550h1m0.797186914s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.106114940 (550h1m0.977197975s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.255964778 (550h1m1.127047823s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282094312 (550h1m1.153177357s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282119982 (550h1m1.153203027s) [seg_pos={0 0}, upload=482242]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:30:01.282360480 (550h1m1.153443525s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection
Dec 28 13:30:01.282371961 (550h1m1.153455006s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282518210 (550h1m1.153601756s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282541183 (550h1m1.153624259s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282731226 (550h1m1.153814271s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282740724 (550h1m1.153823769s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282766573 (550h1m1.153849618s) [seg_pos={0 0}, upload=482242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:30:01.283083446 (550h1m1.154166482s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:30:01.283107071 (550h1m1.154190116s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:30:01.583258308 (550h1m1.454341363s) [upload=482242]: Done uploading segments...
Dec 28 13:30:01.583286001 (550h1m1.454369056s) [upload=482242]: Upload finished. err: <nil>
Duration: 5.812963908s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration:
38.388770179s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:31:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 38.388789856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [...XXXXXX..X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.] duration:
3.517663303s trace:
https://jaeger.dev.storj.io/trace/40de9e068db51b84
FAIL (2025-12-28 13:30:54 UTC)
orbiter warning: context deadline exceeded
Duration: 3.517689153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.........XX.X..........X.XX..XX..X.......XXX......XX......X.] duration:
5.737905017s trace:
https://jaeger.dev.storj.io/trace/2b041200f1e30347
FAIL (2025-12-28 13:30:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 13:29:58.639531113 (550h0m58.510614158s) [upload=482242]: Starting upload
Dec 28 13:29:58.639669457 (550h0m58.510752492s) [upload=482242]: Done starting upload
Dec 28 13:29:58.639790927 (550h0m58.510873972s) [upload=482242]: Uploading segments...
Dec 28 13:29:58.640333441 (550h0m58.511416476s) [upload=482242]: Got next segment. Inline: false
Dec 28 13:29:58.640392624 (550h0m58.511475669s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 13:29:58.640414345 (550h0m58.51149739s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 13:29:58.886569264 (550h0m58.757652299s) [seg_pos={0 0}, upload=482242]: Begin upload segment...
Dec 28 13:29:58.886646741 (550h0m58.757729786s) [seg_pos={0 0}, upload=482242]: Done begin upload segment.
Dec 28 13:29:58.886672791 (550h0m58.757755836s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.887683495 (550h0m58.758766541s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888301993 (550h0m58.759385039s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.888722093 (550h0m58.759805128s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889128457 (550h0m58.760211492s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889580348 (550h0m58.760663393s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.889762865 (550h0m58.7608459s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891199751 (550h0m58.762282796s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.891678813 (550h0m58.762761849s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892085699 (550h0m58.763168734s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.892609716 (550h0m58.763692761s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893015538 (550h0m58.764098573s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.893626081 (550h0m58.764709127s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894201036 (550h0m58.765284071s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.894671662 (550h0m58.765754728s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:29:58.896608850 (550h0m58.767691886s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Uploading piece...
Dec 28 13:30:00.517630309 (550h1m0.388713364s) [upload=482242]: Next returned nil segment
Dec 28 13:30:00.517644185 (550h1m0.38872723s) [upload=482242]: Waiting for error group managing segments...
Dec 28 13:30:00.902991830 (550h1m0.774074865s) [noise=true , address=145.79.227.42:4122 , piece_id=ZMQ33HLEZCMQQXUOUVE32O7TSNMPQQTEFHRYIDZG4XT63ZQ4BP5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.904178460 (550h1m0.775261505s) [noise=true , address=145.79.227.47:4265 , piece_id=4NCUPXHSWLDYHYUVZLMATWGQT2JSRWH6LXKJ6QFT573I3OSM3AKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.913000588 (550h1m0.784083633s) [noise=true , address=145.79.227.19:4077 , piece_id=PP3X2GYF7FNGRHJ2SN7NUW5ADLEFS5KGZTUSZHSYMEFXGUJGLE5Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.917507580 (550h1m0.788590615s) [noise=true , address=145.79.227.3:4150 , piece_id=ICMWLRFBXNHYP5WFURVKD3BJJ5Z5JSMND2HLR2K4VTQEMIGOZGRQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.919104200 (550h1m0.790187235s) [noise=true , address=145.79.227.51:4411 , piece_id=EIKOSWZRLXOUT3MBBGIWZQM5IEV4ZG7GDXH6GKGJJCMHBHNG3BDA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.920957218 (550h1m0.792040253s) [noise=true , address=145.79.227.13:4802 , piece_id=K6I5J4TCIBI5ZDUCFNYNSCHROK5BA6PAX3PGCNAHEXKMKXCHHFKA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.921859647 (550h1m0.792942702s) [noise=true , address=145.79.227.50:4373 , piece_id=HDAADZVLAJ257EWSXPFRF2EC3P2HB7U3HQZMFBP6LWVC4252P3DQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.923841651 (550h1m0.794924706s) [noise=true , address=145.79.227.7:4294 , piece_id=3ARUIOKRVVZ7KLDB6VVEFUURHQLEJBYUY2NOFHZLUJYB5H7GKCQA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.925186592 (550h1m0.796269637s) [noise=true , address=145.79.227.46:4603 , piece_id=FRYHL3TVJOON766ZDAJLNH3TZ4OXRHCNHTCX3MCMFXI2CJ5QT6PQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:00.926103869 (550h1m0.797186914s) [noise=true , address=145.79.227.20:4439 , piece_id=JVGL5PRQX6E6BOACCZ7JK7IHQLQOC354NAIPJZUU5VQHLTZWOEIQ, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.106114940 (550h1m0.977197975s) [noise=true , address=45.91.116.9:4263 , piece_id=JK64MUOQ7Z63JI3VXFBEKZQKJ2ZCT4MJQ32SDIT6N2GXHWP66VUA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.255964778 (550h1m1.127047823s) [noise=true , address=45.91.116.8:4048 , piece_id=HW2EVM4SYGTDAX7CSA5Z4A6XAHFJENHJXMLU5B7RVZE5UGH6WEVA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282094312 (550h1m1.153177357s) [noise=true , address=45.91.116.7:4011 , piece_id=2JLRY6HV4CBLQNJSEW42GMKKP6TMDCYHLGAORXZSCFB5N2IZHXWA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: <nil>
Dec 28 13:30:01.282119982 (550h1m1.153203027s) [seg_pos={0 0}, upload=482242]: Segment reached optimal threshold of 13 pieces.
Dec 28 13:30:01.282360480 (550h1m1.153443525s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57540->45.91.116.12:4227: use of closed network connection
Dec 28 13:30:01.282371961 (550h1m1.153455006s) [noise=true , address=45.91.116.12:4227 , piece_id=BYXJCIWR767TBEUTYSOOA3N5EQUL7YXDYVNMW2BVU3URQ3IER3PA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282518210 (550h1m1.153601756s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282541183 (550h1m1.153624259s) [noise=true , address=45.91.116.5:4298 , piece_id=DPD5LE7E4ENH2EFRJK3RT6N2453MV2SRFPP7MANHAJ642SAZIW6Q, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282731226 (550h1m1.153814271s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 13:30:01.282740724 (550h1m1.153823769s) [noise=true , address=77.90.31.13:4626 , piece_id=GWFZVTSF2VOSQDCR6TYJODZS2RKRMSF7QWRLCHIKEGNRRY6TGNPA, seg_pos={0 0}, upload=482242]: Long tail cancellation due to optimal threshold
Dec 28 13:30:01.282766573 (550h1m1.153849618s) [seg_pos={0 0}, upload=482242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 13:30:01.283083446 (550h1m1.154166482s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:30:01.283107071 (550h1m1.154190116s) [upload=482242]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:30:01.583258308 (550h1m1.454341363s) [upload=482242]: Done uploading segments...
Dec 28 13:30:01.583286001 (550h1m1.454369056s) [upload=482242]: Upload finished. err: <nil>
Duration: 5.73793241s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X........X......] duration:
43.16895118s trace:
https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268
FAIL (2025-12-28 13:30:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece...
Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment
Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments...
Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil>
Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces.
Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold
Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments...
Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil>
Duration: 43.168980987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X.....XX....X...........X....................X.] duration:
13.925145154s trace:
https://jaeger.dev.storj.io/trace/745a9066fe03a6ff
FAIL (2025-12-28 13:30:35 UTC)
orbiter warning: context deadline exceeded
Duration: 13.925176634s
edge
gateway
us-select-1
|