Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX.X..XX..XX...X...X...........X..X.X...........X....XX.XXX.] duration:
5.763789351s trace:
https://jaeger.dev.storj.io/trace/606bd82f4919716a
FAIL (2025-12-26 14:27:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
26 14:26:18.495486994 (502h57m18.366570029s) [upload=441374]: Starting upload
Dec 26 14:26:18.495576194 (502h57m18.366659239s) [upload=441374]: Done starting upload
Dec 26 14:26:18.495627792 (502h57m18.366710837s) [upload=441374]: Uploading segments...
Dec 26 14:26:18.495774342 (502h57m18.366857377s) [upload=441374]: Got next segment. Inline: false
Dec 26 14:26:18.495810450 (502h57m18.366893495s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:26:18.495814598 (502h57m18.366897633s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:26:18.737346699 (502h57m18.608429745s) [seg_pos={0 0}, upload=441374]: Begin upload segment...
Dec 26 14:26:18.737440527 (502h57m18.608523562s) [seg_pos={0 0}, upload=441374]: Done begin upload segment.
Dec 26 14:26:18.737476245 (502h57m18.60855928s) [noise=true , address=145.79.227.49:4339 , piece_id=CPP2VGGUFRY7FNDM7J4SWKRF3SKPD2C6ILSPDRWY4IW53R2HYWCA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.737879884 (502h57m18.608962919s) [noise=true , address=77.90.31.19:4844 , piece_id=7SGTIQMUOTFQ4PUHK6OSFB7MHPELUWY3FCYZRVVGJ6LNUMO7CW4A, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738111035 (502h57m18.60919407s) [noise=true , address=145.79.227.37:4565 , piece_id=6LXKBO6UORBQP4GMX2A77PCPBTENHSCLV4TFS6ZJ745AEL6H3L5Q, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738431315 (502h57m18.60951436s) [noise=true , address=45.91.116.7:4010 , piece_id=EBHYUMBGBICTB4AMAEKNDGBVIO5PFLMLG6SWKGXG4UB3USYI24ZA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738689356 (502h57m18.609772391s) [noise=true , address=145.79.227.41:4087 , piece_id=V7NRAF3KLLO26GPKA6KL6SEU3GC5S3EMPZ7KL6OBJRFE3RP2RR2Q, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738747457 (502h57m18.609830492s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739005558 (502h57m18.610088593s) [noise=true , address=145.79.227.6:4257 , piece_id=PHQU7YHPLBMERZKELZ3AAZPKAZCBCQWXUNH7BM3ZVTFN6ZXABCGA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739193907 (502h57m18.610276942s) [noise=true , address=145.79.227.11:4729 , piece_id=SX3CU7RQS4X5T6YP6MV6L747CSF72GFENQGB2TJ25AKSYNXVYYTQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739309957 (502h57m18.610392993s) [noise=true , address=145.79.227.26:4654 , piece_id=FDZ2CITE7HZ6UAA3IDJ5222M3LPBA5CSFDVCQJ26QX4MTMZL4HSA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739689280 (502h57m18.610772315s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739705431 (502h57m18.610788466s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740067400 (502h57m18.611150435s) [noise=true , address=45.91.116.6:4975 , piece_id=V6K3RCCDQ4JOH64Y2RUASMPBLDN4XCAJ4E2P6CLXQE3XYPP6NUXA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740168463 (502h57m18.611251508s) [noise=true , address=145.79.227.39:4014 , piece_id=EP32WV5DIQGF3XKTPA3AWO5DTPT3OU7VDIQTHVCNCPPT6RH7TLOA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740675698 (502h57m18.611758743s) [noise=true , address=145.79.227.15:4876 , piece_id=SP5IA42GQCP2XRUCNCWLWE4DZP5UGRZRWHDRQ2XVDVCHIHLY6CYQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.741359751 (502h57m18.612442786s) [noise=true , address=145.79.227.13:4802 , piece_id=J7G4J2SDOREY2QKRNAO4PVGDP7ZT6DNWFYE2AVISD7QKTU5QMLUA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.741465061 (502h57m18.612548096s) [noise=true , address=145.79.227.36:4517 , piece_id=S2T4DGRKKE6D3WI4LQZJUA7FYGLBH3X4A3RNCV4MBMVLTNEOPH4A, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:20.452952326 (502h57m20.324035361s) [upload=441374]: Next returned nil segment
Dec 26 14:26:20.452970261 (502h57m20.324053306s) [upload=441374]: Waiting for error group managing segments...
Dec 26 14:26:20.744254178 (502h57m20.615337223s) [noise=true , address=145.79.227.26:4654 , piece_id=FDZ2CITE7HZ6UAA3IDJ5222M3LPBA5CSFDVCQJ26QX4MTMZL4HSA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.750760447 (502h57m20.621843482s) [noise=true , address=145.79.227.11:4729 , piece_id=SX3CU7RQS4X5T6YP6MV6L747CSF72GFENQGB2TJ25AKSYNXVYYTQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.752453371 (502h57m20.623536426s) [noise=true , address=145.79.227.36:4517 , piece_id=S2T4DGRKKE6D3WI4LQZJUA7FYGLBH3X4A3RNCV4MBMVLTNEOPH4A, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.757313866 (502h57m20.628396912s) [noise=true , address=145.79.227.41:4087 , piece_id=V7NRAF3KLLO26GPKA6KL6SEU3GC5S3EMPZ7KL6OBJRFE3RP2RR2Q, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.763767856 (502h57m20.634850901s) [noise=true , address=145.79.227.15:4876 , piece_id=SP5IA42GQCP2XRUCNCWLWE4DZP5UGRZRWHDRQ2XVDVCHIHLY6CYQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.767691417 (502h57m20.638774462s) [noise=true , address=145.79.227.49:4339 , piece_id=CPP2VGGUFRY7FNDM7J4SWKRF3SKPD2C6ILSPDRWY4IW53R2HYWCA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.771013993 (502h57m20.642097038s) [noise=true , address=145.79.227.37:4565 , piece_id=6LXKBO6UORBQP4GMX2A77PCPBTENHSCLV4TFS6ZJ745AEL6H3L5Q, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.776490442 (502h57m20.647573487s) [noise=true , address=145.79.227.13:4802 , piece_id=J7G4J2SDOREY2QKRNAO4PVGDP7ZT6DNWFYE2AVISD7QKTU5QMLUA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.777890778 (502h57m20.648973813s) [noise=true , address=145.79.227.6:4257 , piece_id=PHQU7YHPLBMERZKELZ3AAZPKAZCBCQWXUNH7BM3ZVTFN6ZXABCGA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.778020245 (502h57m20.64910328s) [noise=true , address=145.79.227.39:4014 , piece_id=EP32WV5DIQGF3XKTPA3AWO5DTPT3OU7VDIQTHVCNCPPT6RH7TLOA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.030338154 (502h57m20.901421189s) [noise=true , address=45.91.116.7:4010 , piece_id=EBHYUMBGBICTB4AMAEKNDGBVIO5PFLMLG6SWKGXG4UB3USYI24ZA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.031018089 (502h57m20.902101134s) [noise=true , address=45.91.116.6:4975 , piece_id=V6K3RCCDQ4JOH64Y2RUASMPBLDN4XCAJ4E2P6CLXQE3XYPP6NUXA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.036144861 (502h57m20.907227896s) [noise=true , address=77.90.31.19:4844 , piece_id=7SGTIQMUOTFQ4PUHK6OSFB7MHPELUWY3FCYZRVVGJ6LNUMO7CW4A, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.036165671 (502h57m20.907248716s) [seg_pos={0 0}, upload=441374]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:26:21.036359079 (502h57m20.907442124s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 26 14:26:21.036365130 (502h57m20.907448175s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036652608 (502h57m20.907735643s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection
Dec 26 14:26:21.036694979 (502h57m20.907778024s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036816780 (502h57m20.907899825s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 26 14:26:21.036826028 (502h57m20.907909154s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036847048 (502h57m20.907930093s) [seg_pos={0 0}, upload=441374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:26:21.036948291 (502h57m20.908031336s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:26:21.036955083 (502h57m20.908038129s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:26:21.326562880 (502h57m21.197645925s) [upload=441374]: Done uploading segments...
Dec 26 14:26:21.326587657 (502h57m21.197670692s) [upload=441374]: Upload finished. err: <nil>
Duration: 5.763819918s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXXX] duration:
5.001402497s trace:
https://jaeger.dev.storj.io/trace/2db34bf3e9a94036
FAIL (2025-12-26 14:27:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001427613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.....] duration:
58.562404907s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:27:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 58.562425836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXX.X..XX..XX...X...X...........X..X.X...........X....XX.XXX] duration:
30.003743964s trace:
https://jaeger.dev.storj.io/trace/606bd82f4919716a
FAIL (2025-12-26 14:26:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
26 14:26:18.495486994 (502h57m18.366570029s) [upload=441374]: Starting upload
Dec 26 14:26:18.495576194 (502h57m18.366659239s) [upload=441374]: Done starting upload
Dec 26 14:26:18.495627792 (502h57m18.366710837s) [upload=441374]: Uploading segments...
Dec 26 14:26:18.495774342 (502h57m18.366857377s) [upload=441374]: Got next segment. Inline: false
Dec 26 14:26:18.495810450 (502h57m18.366893495s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:26:18.495814598 (502h57m18.366897633s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:26:18.737346699 (502h57m18.608429745s) [seg_pos={0 0}, upload=441374]: Begin upload segment...
Dec 26 14:26:18.737440527 (502h57m18.608523562s) [seg_pos={0 0}, upload=441374]: Done begin upload segment.
Dec 26 14:26:18.737476245 (502h57m18.60855928s) [noise=true , address=145.79.227.49:4339 , piece_id=CPP2VGGUFRY7FNDM7J4SWKRF3SKPD2C6ILSPDRWY4IW53R2HYWCA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.737879884 (502h57m18.608962919s) [noise=true , address=77.90.31.19:4844 , piece_id=7SGTIQMUOTFQ4PUHK6OSFB7MHPELUWY3FCYZRVVGJ6LNUMO7CW4A, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738111035 (502h57m18.60919407s) [noise=true , address=145.79.227.37:4565 , piece_id=6LXKBO6UORBQP4GMX2A77PCPBTENHSCLV4TFS6ZJ745AEL6H3L5Q, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738431315 (502h57m18.60951436s) [noise=true , address=45.91.116.7:4010 , piece_id=EBHYUMBGBICTB4AMAEKNDGBVIO5PFLMLG6SWKGXG4UB3USYI24ZA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738689356 (502h57m18.609772391s) [noise=true , address=145.79.227.41:4087 , piece_id=V7NRAF3KLLO26GPKA6KL6SEU3GC5S3EMPZ7KL6OBJRFE3RP2RR2Q, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.738747457 (502h57m18.609830492s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739005558 (502h57m18.610088593s) [noise=true , address=145.79.227.6:4257 , piece_id=PHQU7YHPLBMERZKELZ3AAZPKAZCBCQWXUNH7BM3ZVTFN6ZXABCGA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739193907 (502h57m18.610276942s) [noise=true , address=145.79.227.11:4729 , piece_id=SX3CU7RQS4X5T6YP6MV6L747CSF72GFENQGB2TJ25AKSYNXVYYTQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739309957 (502h57m18.610392993s) [noise=true , address=145.79.227.26:4654 , piece_id=FDZ2CITE7HZ6UAA3IDJ5222M3LPBA5CSFDVCQJ26QX4MTMZL4HSA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739689280 (502h57m18.610772315s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.739705431 (502h57m18.610788466s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740067400 (502h57m18.611150435s) [noise=true , address=45.91.116.6:4975 , piece_id=V6K3RCCDQ4JOH64Y2RUASMPBLDN4XCAJ4E2P6CLXQE3XYPP6NUXA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740168463 (502h57m18.611251508s) [noise=true , address=145.79.227.39:4014 , piece_id=EP32WV5DIQGF3XKTPA3AWO5DTPT3OU7VDIQTHVCNCPPT6RH7TLOA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.740675698 (502h57m18.611758743s) [noise=true , address=145.79.227.15:4876 , piece_id=SP5IA42GQCP2XRUCNCWLWE4DZP5UGRZRWHDRQ2XVDVCHIHLY6CYQ, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.741359751 (502h57m18.612442786s) [noise=true , address=145.79.227.13:4802 , piece_id=J7G4J2SDOREY2QKRNAO4PVGDP7ZT6DNWFYE2AVISD7QKTU5QMLUA, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:18.741465061 (502h57m18.612548096s) [noise=true , address=145.79.227.36:4517 , piece_id=S2T4DGRKKE6D3WI4LQZJUA7FYGLBH3X4A3RNCV4MBMVLTNEOPH4A, seg_pos={0 0}, upload=441374]: Uploading piece...
Dec 26 14:26:20.452952326 (502h57m20.324035361s) [upload=441374]: Next returned nil segment
Dec 26 14:26:20.452970261 (502h57m20.324053306s) [upload=441374]: Waiting for error group managing segments...
Dec 26 14:26:20.744254178 (502h57m20.615337223s) [noise=true , address=145.79.227.26:4654 , piece_id=FDZ2CITE7HZ6UAA3IDJ5222M3LPBA5CSFDVCQJ26QX4MTMZL4HSA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.750760447 (502h57m20.621843482s) [noise=true , address=145.79.227.11:4729 , piece_id=SX3CU7RQS4X5T6YP6MV6L747CSF72GFENQGB2TJ25AKSYNXVYYTQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.752453371 (502h57m20.623536426s) [noise=true , address=145.79.227.36:4517 , piece_id=S2T4DGRKKE6D3WI4LQZJUA7FYGLBH3X4A3RNCV4MBMVLTNEOPH4A, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.757313866 (502h57m20.628396912s) [noise=true , address=145.79.227.41:4087 , piece_id=V7NRAF3KLLO26GPKA6KL6SEU3GC5S3EMPZ7KL6OBJRFE3RP2RR2Q, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.763767856 (502h57m20.634850901s) [noise=true , address=145.79.227.15:4876 , piece_id=SP5IA42GQCP2XRUCNCWLWE4DZP5UGRZRWHDRQ2XVDVCHIHLY6CYQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.767691417 (502h57m20.638774462s) [noise=true , address=145.79.227.49:4339 , piece_id=CPP2VGGUFRY7FNDM7J4SWKRF3SKPD2C6ILSPDRWY4IW53R2HYWCA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.771013993 (502h57m20.642097038s) [noise=true , address=145.79.227.37:4565 , piece_id=6LXKBO6UORBQP4GMX2A77PCPBTENHSCLV4TFS6ZJ745AEL6H3L5Q, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.776490442 (502h57m20.647573487s) [noise=true , address=145.79.227.13:4802 , piece_id=J7G4J2SDOREY2QKRNAO4PVGDP7ZT6DNWFYE2AVISD7QKTU5QMLUA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.777890778 (502h57m20.648973813s) [noise=true , address=145.79.227.6:4257 , piece_id=PHQU7YHPLBMERZKELZ3AAZPKAZCBCQWXUNH7BM3ZVTFN6ZXABCGA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:20.778020245 (502h57m20.64910328s) [noise=true , address=145.79.227.39:4014 , piece_id=EP32WV5DIQGF3XKTPA3AWO5DTPT3OU7VDIQTHVCNCPPT6RH7TLOA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.030338154 (502h57m20.901421189s) [noise=true , address=45.91.116.7:4010 , piece_id=EBHYUMBGBICTB4AMAEKNDGBVIO5PFLMLG6SWKGXG4UB3USYI24ZA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.031018089 (502h57m20.902101134s) [noise=true , address=45.91.116.6:4975 , piece_id=V6K3RCCDQ4JOH64Y2RUASMPBLDN4XCAJ4E2P6CLXQE3XYPP6NUXA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.036144861 (502h57m20.907227896s) [noise=true , address=77.90.31.19:4844 , piece_id=7SGTIQMUOTFQ4PUHK6OSFB7MHPELUWY3FCYZRVVGJ6LNUMO7CW4A, seg_pos={0 0}, upload=441374]: Done uploading piece. err: <nil>
Dec 26 14:26:21.036165671 (502h57m20.907248716s) [seg_pos={0 0}, upload=441374]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:26:21.036359079 (502h57m20.907442124s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 26 14:26:21.036365130 (502h57m20.907448175s) [noise=true , address=145.79.227.33:4870 , piece_id=JUVSBVVUX7KK4LTVTMHAU2V5E7YHHT5YUSATLVQYY6RSVILRGCTA, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036652608 (502h57m20.907735643s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57688->45.91.116.9:4264: use of closed network connection
Dec 26 14:26:21.036694979 (502h57m20.907778024s) [noise=true , address=45.91.116.9:4264 , piece_id=YXD7P2FQSCDBO2YBPGKD7FIKEFPS3MXSS6J2JLIL7RKI3CSJTCMA, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036816780 (502h57m20.907899825s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 26 14:26:21.036826028 (502h57m20.907909154s) [noise=true , address=45.91.116.13:4154 , piece_id=OUZ3GOIJGEPJHUQ2KK54FGYEJGATQ7C43UKWFRL7G6VM4NYQLBWQ, seg_pos={0 0}, upload=441374]: Long tail cancellation due to optimal threshold
Dec 26 14:26:21.036847048 (502h57m20.907930093s) [seg_pos={0 0}, upload=441374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:26:21.036948291 (502h57m20.908031336s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:26:21.036955083 (502h57m20.908038129s) [upload=441374]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:26:21.326562880 (502h57m21.197645925s) [upload=441374]: Done uploading segments...
Dec 26 14:26:21.326587657 (502h57m21.197670692s) [upload=441374]: Upload finished. err: <nil>
Duration: 30.00377275s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
8.695813029s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:26:43 UTC)
orbiter warning: context deadline exceeded
Duration: 8.695837514s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [X.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XXX] duration:
5.000686412s trace:
https://jaeger.dev.storj.io/trace/4cb724c80a9c7bf
FAIL (2025-12-26 14:26:41 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000708143s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
14.360234782s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:26:29 UTC)
orbiter warning: context deadline exceeded
Duration: 14.360310926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
14.380522662s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:26:28 UTC)
orbiter warning: context deadline exceeded
Duration: 14.380582406s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX....] duration:
59.723326008s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:26:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 59.723378528s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.XX] duration:
5.001666858s trace:
https://jaeger.dev.storj.io/trace/497e6af061a44b0c
FAIL (2025-12-26 14:26:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001690862s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
19.908842525s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:25:51 UTC)
orbiter warning: context deadline exceeded
Duration: 19.908869397s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
15.009987269s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:25:39 UTC)
orbiter warning: context deadline exceeded
Duration: 15.010015342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXXX.X..XX..XX...X...X...........X..X.X...........X....XX.XX] duration:
30.002920676s trace:
https://jaeger.dev.storj.io/trace/1ba101c8f536d8c3
FAIL (2025-12-26 14:25:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7MDDPPVANKH77U23G2FWI6FGYSZHSTRT6G4VIZVKZFMYMT2DMNTQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: F52QSF6TCII2RMJYQUXDTJT3OTLI3PFXCBUPZDMTSNIRCYDSDT5Q) context deadline exceeded
======LOGS======
Dec 26 14:25:05.053547017 (502h56m4.924630052s) [upload=441357]: Starting upload
Dec 26 14:25:05.053624394 (502h56m4.924707439s) [upload=441357]: Done starting upload
Dec 26 14:25:05.053930848 (502h56m4.925013893s) [upload=441357]: Uploading segments...
Dec 26 14:25:05.053989580 (502h56m4.925072625s) [upload=441357]: Got next segment. Inline: false
Dec 26 14:25:05.054024637 (502h56m4.925107672s) [upload=441357]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:25:05.054030458 (502h56m4.925113503s) [upload=441357]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:25:05.308191227 (502h56m5.179274262s) [seg_pos={0 0}, upload=441357]: Begin upload segment...
Dec 26 14:25:05.308238226 (502h56m5.179321271s) [seg_pos={0 0}, upload=441357]: Done begin upload segment.
Dec 26 14:25:05.308264306 (502h56m5.179347351s) [noise=true , address=77.90.31.14:4661 , piece_id=F52QSF6TCII2RMJYQUXDTJT3OTLI3PFXCBUPZDMTSNIRCYDSDT5Q, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.308324110 (502h56m5.179407155s) [noise=true , address=145.79.227.41:4087 , piece_id=FHPD6U5N6CUALSDRZY2WR764NAZRFP5G2AVAA3MNTI3H3QD7LWMQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.308853618 (502h56m5.179936663s) [noise=true , address=145.79.227.35:4482 , piece_id=G6KQ5AA7C2FKLGUWKCZUOPOFRUUTCVT7DR6OIJ5GUYQ6TFYHVA2A, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.309388646 (502h56m5.180471691s) [noise=true , address=77.90.31.17:4770 , piece_id=KXR74XLKVVNWZWL5PUHOT7ZORVEBRTFI7SFMOS54OSDRNHK5MYNQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.309915409 (502h56m5.180998444s) [noise=true , address=45.91.116.9:4262 , piece_id=N4NKN72UF6C7LWLPNZQV6PS2KHJNTVSUSUGL4ARIR445BO2G4WLA, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.310289662 (502h56m5.181372707s) [noise=true , address=45.91.116.6:4976 , piece_id=7MDDPPVANKH77U23G2FWI6FGYSZHSTRT6G4VIZVKZFMYMT2DMNTQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.310694193 (502h56m5.181777238s) [noise=true , address=145.79.227.31:4799 , piece_id=O4ZIDTJCP542JSIYKFCKT7QQ4WRYF3SXUKHKY2XU5FKFWR54ZN6A, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.311098372 (502h56m5.182181407s) [noise=true , address=145.79.227.51:4409 , piece_id=PDOX4AN43PY5O5SLGSIV32UTHGOAJVR3Q7XZUVFF4Z3CEVEUUG5Q, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.311549761 (502h56m5.182632806s) [noise=true , address=77.90.31.11:4553 , piece_id=6GNSUVWV7IDAP4LXY4VEJLLTVIMUCDKVWK3KWY6ION7NG3QZT3IQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.312233674 (502h56m5.183316719s) [noise=true , address=145.79.227.36:4518 , piece_id=7LDW4BRU5PZLTR6GMVY442EQDBNAD5EIOYKDCFUIBYBRDVAWNALA, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.313714073 (502h56m5.184797119s) [noise=true , address=145.79.227.16:4910 , piece_id=G2O7REJSEPXN45IIMWZJQVPHMOQ4BBPDTV5AQBHUHLMODBPIESCQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.314409277 (502h56m5.185492332s) [noise=true , address=145.79.227.39:4014 , piece_id=HWIHPIH3JGYSQK75GEG4HVKOY3EPQ2F6OUZ5XWFIBXM6MOUEGQNA, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.316229994 (502h56m5.187313039s) [noise=true , address=145.79.227.44:4193 , piece_id=RYVCMDMSYJAGJJJRZIWRJRTDL2RN7P6U3GU5DWRDHRRWH36XIYBQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.318250792 (502h56m5.189333837s) [noise=true , address=145.79.227.5:4222 , piece_id=B43TTH4SIKNVSMNVXDUCY5H3H5W4JBU536KSG6YLJLFZ3GAAKSIQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.320424591 (502h56m5.191507636s) [noise=true , address=145.79.227.49:4337 , piece_id=PMGVE2T7XLQ6Q5GCK7DGWHL7VE2EEPHRWWHHHRFX4MZLZ4HPXTLQ, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:05.322316995 (502h56m5.19340004s) [noise=true , address=145.79.227.47:4265 , piece_id=RUOH3SHBUMHNHP7CAFDGXWBVLKXKD6Y32GKOL5RFLABGTKXSCH3Q, seg_pos={0 0}, upload=441357]: Uploading piece...
Dec 26 14:25:06.445427642 (502h56m6.316510677s) [noise=true , address=77.90.31.11:4553 , piece_id=6GNSUVWV7IDAP4LXY4VEJLLTVIMUCDKVWK3KWY6ION7NG3QZT3IQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000
Dec 26 14:25:07.002800643 (502h56m6.873883698s) [upload=441357]: Next returned nil segment
Dec 26 14:25:07.002830769 (502h56m6.873913814s) [upload=441357]: Waiting for error group managing segments...
Dec 26 14:25:07.321872713 (502h56m7.192955758s) [noise=true , address=145.79.227.49:4337 , piece_id=PMGVE2T7XLQ6Q5GCK7DGWHL7VE2EEPHRWWHHHRFX4MZLZ4HPXTLQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.322843031 (502h56m7.193926076s) [noise=true , address=145.79.227.51:4409 , piece_id=PDOX4AN43PY5O5SLGSIV32UTHGOAJVR3Q7XZUVFF4Z3CEVEUUG5Q, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.326615215 (502h56m7.19769826s) [noise=true , address=145.79.227.31:4799 , piece_id=O4ZIDTJCP542JSIYKFCKT7QQ4WRYF3SXUKHKY2XU5FKFWR54ZN6A, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.326695768 (502h56m7.197778813s) [noise=true , address=145.79.227.16:4910 , piece_id=G2O7REJSEPXN45IIMWZJQVPHMOQ4BBPDTV5AQBHUHLMODBPIESCQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.327181793 (502h56m7.198264848s) [noise=true , address=145.79.227.36:4518 , piece_id=7LDW4BRU5PZLTR6GMVY442EQDBNAD5EIOYKDCFUIBYBRDVAWNALA, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.331508963 (502h56m7.202592009s) [noise=true , address=145.79.227.39:4014 , piece_id=HWIHPIH3JGYSQK75GEG4HVKOY3EPQ2F6OUZ5XWFIBXM6MOUEGQNA, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.335306485 (502h56m7.20638953s) [noise=true , address=145.79.227.35:4482 , piece_id=G6KQ5AA7C2FKLGUWKCZUOPOFRUUTCVT7DR6OIJ5GUYQ6TFYHVA2A, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.345401467 (502h56m7.216484512s) [noise=true , address=145.79.227.41:4087 , piece_id=FHPD6U5N6CUALSDRZY2WR764NAZRFP5G2AVAA3MNTI3H3QD7LWMQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.349103175 (502h56m7.220186221s) [noise=true , address=145.79.227.5:4222 , piece_id=B43TTH4SIKNVSMNVXDUCY5H3H5W4JBU536KSG6YLJLFZ3GAAKSIQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.370164560 (502h56m7.241247605s) [noise=true , address=145.79.227.47:4265 , piece_id=RUOH3SHBUMHNHP7CAFDGXWBVLKXKD6Y32GKOL5RFLABGTKXSCH3Q, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.533035487 (502h56m7.404118532s) [noise=true , address=77.90.31.17:4770 , piece_id=KXR74XLKVVNWZWL5PUHOT7ZORVEBRTFI7SFMOS54OSDRNHK5MYNQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:07.662995204 (502h56m7.534078249s) [noise=true , address=77.90.31.14:4661 , piece_id=F52QSF6TCII2RMJYQUXDTJT3OTLI3PFXCBUPZDMTSNIRCYDSDT5Q, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:08.044897716 (502h56m7.915980762s) [noise=true , address=45.91.116.6:4976 , piece_id=7MDDPPVANKH77U23G2FWI6FGYSZHSTRT6G4VIZVKZFMYMT2DMNTQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: <nil>
Dec 26 14:25:08.044920290 (502h56m7.916003335s) [seg_pos={0 0}, upload=441357]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:25:08.045252482 (502h56m7.916335527s) [noise=true , address=145.79.227.44:4193 , piece_id=RYVCMDMSYJAGJJJRZIWRJRTDL2RN7P6U3GU5DWRDHRRWH36XIYBQ, seg_pos={0 0}, upload=441357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 26 14:25:08.045263263 (502h56m7.916346308s) [noise=true , address=145.79.227.44:4193 , piece_id=RYVCMDMSYJAGJJJRZIWRJRTDL2RN7P6U3GU5DWRDHRRWH36XIYBQ, seg_pos={0 0}, upload=441357]: Long tail cancellation due to optimal threshold
Dec 26 14:25:08.045715855 (502h56m7.91679889s) [noise=true , address=45.91.116.9:4262 , piece_id=N4NKN72UF6C7LWLPNZQV6PS2KHJNTVSUSUGL4ARIR445BO2G4WLA, seg_pos={0 0}, upload=441357]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: read tcp 188.166.225.170:58888->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:25:08.045726705 (502h56m7.91680974s) [noise=true , address=45.91.116.9:4262 , piece_id=N4NKN72UF6C7LWLPNZQV6PS2KHJNTVSUSUGL4ARIR445BO2G4WLA, seg_pos={0 0}, upload=441357]: Long tail cancellation due to optimal threshold
Dec 26 14:25:08.045747644 (502h56m7.91683069s) [seg_pos={0 0}, upload=441357]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:25:08.046655735 (502h56m7.91773877s) [upload=441357]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:25:08.046671314 (502h56m7.917754359s) [upload=441357]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:25:08.337117386 (502h56m8.208200461s) [upload=441357]: Done uploading segments...
Dec 26 14:25:08.337239518 (502h56m8.208322593s) [upload=441357]: Upload finished. err: <nil>
Duration: 30.002936967s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [.XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.X] duration:
5.00131658s trace:
https://jaeger.dev.storj.io/trace/30d300fb853bb351
FAIL (2025-12-26 14:25:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001337079s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
16.212990419s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:25:20 UTC)
orbiter warning: context deadline exceeded
Duration: 16.213008262s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX...] duration:
46.124030284s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:25:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 46.124057507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXXXX.X..XX..XX...X...X...........X..X.X...........X....XX.X] duration:
30.007359668s trace:
https://jaeger.dev.storj.io/trace/276970285154e53b
FAIL (2025-12-26 14:25:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 7VPI43MT4UIPEWQ4KI52TG4IFEAJP7HJ45EXZBTY5XYFZLSN4VOQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: WBB3CQASPVA6XB7Q7ZXE5JDQ64OXANBALHPUWK27RPKOW33QS3JQ) context deadline exceeded
======LOGS======
Dec 26 14:24:32.823738531 (502h55m32.694821566s) [upload=441349]: Starting upload
Dec 26 14:24:32.823801129 (502h55m32.694884164s) [upload=441349]: Done starting upload
Dec 26 14:24:32.823943771 (502h55m32.695026806s) [upload=441349]: Uploading segments...
Dec 26 14:24:32.824094088 (502h55m32.695177123s) [upload=441349]: Got next segment. Inline: false
Dec 26 14:24:32.824311521 (502h55m32.695394566s) [upload=441349]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:24:32.824317052 (502h55m32.695400097s) [upload=441349]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:24:33.078674375 (502h55m32.94975742s) [seg_pos={0 0}, upload=441349]: Begin upload segment...
Dec 26 14:24:33.078722847 (502h55m32.949805892s) [seg_pos={0 0}, upload=441349]: Done begin upload segment.
Dec 26 14:24:33.078791518 (502h55m32.949874563s) [noise=true , address=77.90.31.15:4700 , piece_id=7VPI43MT4UIPEWQ4KI52TG4IFEAJP7HJ45EXZBTY5XYFZLSN4VOQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.078803190 (502h55m32.949886225s) [noise=true , address=145.79.227.15:4875 , piece_id=6SQLS5JBAZKT2KKJXVN36FILFGYKN2ACZWG4W5NHRGLOMVG6VITA, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.079438440 (502h55m32.950521485s) [noise=true , address=145.79.227.18:4007 , piece_id=KUITR77O4QCDBKX73I6UHBTCGPF55NW7SLKW7HBCUO2SHXGVFA5A, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.079688857 (502h55m32.950771892s) [noise=true , address=77.90.31.10:4520 , piece_id=XMZY4N73PFZR7MTVOWUJLP3NWBS7KZT7CDWBPJNAK3TV6NC3ZI2A, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.079929515 (502h55m32.95101255s) [noise=true , address=145.79.227.50:4375 , piece_id=EHA7UBDRGS4L5D5EAMIT2ZQOFYCSMCCGJM7FCFINJUIG7IQFYG4Q, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.080043642 (502h55m32.951126687s) [noise=true , address=145.79.227.37:4566 , piece_id=JKPLUNEYC2PQ4IWNHWIW4T2FX6ULHPBPTTMQH55V5SQ5VQ3BWNFQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.080537332 (502h55m32.951620367s) [noise=true , address=45.91.116.9:4263 , piece_id=WBB3CQASPVA6XB7Q7ZXE5JDQ64OXANBALHPUWK27RPKOW33QS3JQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.080696205 (502h55m32.95177924s) [noise=true , address=77.90.31.17:4773 , piece_id=PRWDBUWJKEM7S3XS3DKW35YDP656BLNEIVSUV5BDMKAMJWMAB3OQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.081102288 (502h55m32.952185334s) [noise=true , address=145.79.227.36:4519 , piece_id=3QCNQLOZUG6ZGNUPR4GM636C6HU6RJ7FKLZUFLLUPHOBCSMXBDQA, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.082416079 (502h55m32.953499124s) [noise=true , address=145.79.227.38:4977 , piece_id=43K4OKTUN6EENXLYSLJ7WEGB3MGXW3OQ4K3ONRWNY25W4TQVRFWQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.082995563 (502h55m32.954078608s) [noise=true , address=145.79.227.46:4602 , piece_id=ZHS5O75NH2RB5A452CQ4PWV2G7BS5U3B4M7R76MEK44SS3WPJCRA, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.083552744 (502h55m32.954635779s) [noise=true , address=145.79.227.14:4837 , piece_id=YZQDHY75X6CRP26TBKUKTQGZ3YB34ADMNPM6RW4W3VADQMPZ6P3A, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.084091961 (502h55m32.955174997s) [noise=true , address=77.90.31.11:4553 , piece_id=WPM265ZIHMGG7P5OEWYNUBDELWP4FAX5PHYJEVAQBNGWTCQ4BAUQ, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.084562186 (502h55m32.955645221s) [noise=true , address=145.79.227.11:4730 , piece_id=RQWFGVS7T2VSEWVTEFXBPY7EFTQ2IF2RDJOXCVXUBV36RXDTCJ2A, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.086391640 (502h55m32.957474675s) [noise=true , address=145.79.227.12:4767 , piece_id=ITBQIS3I4CZYIOCX2H46BJER3WS3EEWV4NJMV5R2KSQKSZEFUYTA, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:33.087077245 (502h55m32.95816028s) [noise=true , address=145.79.227.5:4223 , piece_id=J7WKDU5Z2XWAPL5ISFTS4RUTJCASORYMR2YWPBCCFOZCTWUVMLZA, seg_pos={0 0}, upload=441349]: Uploading piece...
Dec 26 14:24:34.303328181 (502h55m34.174411226s) [noise=true , address=77.90.31.11:4553 , piece_id=WPM265ZIHMGG7P5OEWYNUBDELWP4FAX5PHYJEVAQBNGWTCQ4BAUQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000
Dec 26 14:24:34.797271389 (502h55m34.668354434s) [upload=441349]: Next returned nil segment
Dec 26 14:24:34.797291487 (502h55m34.668374522s) [upload=441349]: Waiting for error group managing segments...
Dec 26 14:24:35.079902300 (502h55m34.950985345s) [noise=true , address=145.79.227.15:4875 , piece_id=6SQLS5JBAZKT2KKJXVN36FILFGYKN2ACZWG4W5NHRGLOMVG6VITA, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.080622081 (502h55m34.951705126s) [noise=true , address=145.79.227.18:4007 , piece_id=KUITR77O4QCDBKX73I6UHBTCGPF55NW7SLKW7HBCUO2SHXGVFA5A, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.091445090 (502h55m34.962528135s) [noise=true , address=145.79.227.36:4519 , piece_id=3QCNQLOZUG6ZGNUPR4GM636C6HU6RJ7FKLZUFLLUPHOBCSMXBDQA, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.092310357 (502h55m34.963393392s) [noise=true , address=145.79.227.50:4375 , piece_id=EHA7UBDRGS4L5D5EAMIT2ZQOFYCSMCCGJM7FCFINJUIG7IQFYG4Q, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.093540499 (502h55m34.964623544s) [noise=true , address=145.79.227.5:4223 , piece_id=J7WKDU5Z2XWAPL5ISFTS4RUTJCASORYMR2YWPBCCFOZCTWUVMLZA, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.094112819 (502h55m34.965195864s) [noise=true , address=145.79.227.38:4977 , piece_id=43K4OKTUN6EENXLYSLJ7WEGB3MGXW3OQ4K3ONRWNY25W4TQVRFWQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.106370830 (502h55m34.977453875s) [noise=true , address=145.79.227.37:4566 , piece_id=JKPLUNEYC2PQ4IWNHWIW4T2FX6ULHPBPTTMQH55V5SQ5VQ3BWNFQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.110523258 (502h55m34.981606303s) [noise=true , address=145.79.227.14:4837 , piece_id=YZQDHY75X6CRP26TBKUKTQGZ3YB34ADMNPM6RW4W3VADQMPZ6P3A, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.110724982 (502h55m34.981808017s) [noise=true , address=145.79.227.11:4730 , piece_id=RQWFGVS7T2VSEWVTEFXBPY7EFTQ2IF2RDJOXCVXUBV36RXDTCJ2A, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.111433320 (502h55m34.982516355s) [noise=true , address=145.79.227.46:4602 , piece_id=ZHS5O75NH2RB5A452CQ4PWV2G7BS5U3B4M7R76MEK44SS3WPJCRA, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.113463647 (502h55m34.984546682s) [noise=true , address=145.79.227.12:4767 , piece_id=ITBQIS3I4CZYIOCX2H46BJER3WS3EEWV4NJMV5R2KSQKSZEFUYTA, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.304646393 (502h55m35.175729428s) [noise=true , address=45.91.116.9:4263 , piece_id=WBB3CQASPVA6XB7Q7ZXE5JDQ64OXANBALHPUWK27RPKOW33QS3JQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.393836387 (502h55m35.264919422s) [noise=true , address=77.90.31.15:4700 , piece_id=7VPI43MT4UIPEWQ4KI52TG4IFEAJP7HJ45EXZBTY5XYFZLSN4VOQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: <nil>
Dec 26 14:24:35.393855764 (502h55m35.264938809s) [seg_pos={0 0}, upload=441349]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:24:35.394071515 (502h55m35.26515456s) [noise=true , address=77.90.31.17:4773 , piece_id=PRWDBUWJKEM7S3XS3DKW35YDP656BLNEIVSUV5BDMKAMJWMAB3OQ, seg_pos={0 0}, upload=441349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:24:35.394078618 (502h55m35.265161653s) [noise=true , address=77.90.31.17:4773 , piece_id=PRWDBUWJKEM7S3XS3DKW35YDP656BLNEIVSUV5BDMKAMJWMAB3OQ, seg_pos={0 0}, upload=441349]: Long tail cancellation due to optimal threshold
Dec 26 14:24:35.394298918 (502h55m35.265381964s) [noise=true , address=77.90.31.10:4520 , piece_id=XMZY4N73PFZR7MTVOWUJLP3NWBS7KZT7CDWBPJNAK3TV6NC3ZI2A, seg_pos={0 0}, upload=441349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: rpc: dial tcp 77.90.31.10:4520: operation was canceled; context canceled
Dec 26 14:24:35.394305371 (502h55m35.265388406s) [noise=true , address=77.90.31.10:4520 , piece_id=XMZY4N73PFZR7MTVOWUJLP3NWBS7KZT7CDWBPJNAK3TV6NC3ZI2A, seg_pos={0 0}, upload=441349]: Long tail cancellation due to optimal threshold
Dec 26 14:24:35.394321411 (502h55m35.265404456s) [seg_pos={0 0}, upload=441349]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:24:35.394433263 (502h55m35.265516309s) [upload=441349]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:24:35.394439686 (502h55m35.265522732s) [upload=441349]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:24:35.688616502 (502h55m35.559699537s) [upload=441349]: Done uploading segments...
Dec 26 14:24:35.688636269 (502h55m35.559719314s) [upload=441349]: Upload finished. err: <nil>
Duration: 30.007384976s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX.] duration:
4.273522538s trace:
https://jaeger.dev.storj.io/trace/1fa184f1443be84b
FAIL (2025-12-26 14:24:58 UTC)
orbiter warning: context deadline exceeded
Duration: 4.273561243s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
14.767666957s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:24:51 UTC)
orbiter warning: context deadline exceeded
Duration: 14.767691836s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
11.670650617s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:24:32 UTC)
orbiter warning: context deadline exceeded
Duration: 11.670673941s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
14.791198061s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:24:22 UTC)
orbiter warning: context deadline exceeded
Duration: 14.791252173s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX..] duration:
45.016316469s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:24:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 45.016389167s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
17.153018046s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:24:15 UTC)
orbiter warning: context deadline exceeded
Duration: 17.153060227s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [X..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXXX] duration:
5.000684695s trace:
https://jaeger.dev.storj.io/trace/1fa184f1443be84b
FAIL (2025-12-26 14:24:14 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000711678s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXXX] duration:
5.000821798s trace:
https://jaeger.dev.storj.io/trace/49a4c59abab3a2ec
FAIL (2025-12-26 14:23:39 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000844681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX.] duration:
59.024035279s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:23:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 59.024051421s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XXXXX.X..XX..XX...X...X...........X..X.X...........X....XX.] duration:
5.820860733s trace:
https://jaeger.dev.storj.io/trace/296cad4c0b00743d
FAIL (2025-12-26 14:23:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ) context deadline exceeded
======LOGS======
ing segments...
Dec 26 14:22:54.002229510 (502h53m53.873312545s) [upload=441327]: Got next segment. Inline: false
Dec 26 14:22:54.002280497 (502h53m53.873363532s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:22:54.002289053 (502h53m53.873372089s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:22:54.256285839 (502h53m54.127368874s) [seg_pos={0 0}, upload=441327]: Begin upload segment...
Dec 26 14:22:54.256338528 (502h53m54.127421574s) [seg_pos={0 0}, upload=441327]: Done begin upload segment.
Dec 26 14:22:54.256358307 (502h53m54.127441352s) [noise=true , address=45.91.116.7:4010 , piece_id=VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.256369878 (502h53m54.127452923s) [noise=true , address=145.79.227.43:4159 , piece_id=MA4AGRGCXRSJK7IQ2ADMKF7IA5KMN65HOF6PN2DN6ZEH7WREWKLQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257039414 (502h53m54.128122449s) [noise=true , address=77.90.31.21:4880 , piece_id=D7XLYLXHWB6LOC2F7ZETQDBGBRWL44KCXCQ4ONXC2PONOQQT6P4Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257520890 (502h53m54.128603935s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257943105 (502h53m54.12902615s) [noise=true , address=145.79.227.19:4078 , piece_id=AMIDTE37DZPBL2NBJW7F4D7ZL4KUCW5QHV66VG3CSXOYRVMY3PAA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.259271624 (502h53m54.130354669s) [noise=true , address=145.79.227.15:4876 , piece_id=CJJRQUTBIKPIHFMOG72UTQL5HGNE5KOT6UIMEE37AWIR23HQOPDA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.259721591 (502h53m54.130804626s) [noise=true , address=145.79.227.5:4222 , piece_id=ALTOF6FDW4AJZDSTMGMIVJO74WCAESWPYDNWXOJSJYETXBY6DWSA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.260150327 (502h53m54.131233362s) [noise=true , address=145.79.227.49:4337 , piece_id=JFFJUSVLZK3EOBQA5U43FLHBLFFRPMVCTMTQFJABLLPJVUNP25OA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.261389848 (502h53m54.132472893s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.261633412 (502h53m54.132716467s) [noise=true , address=45.91.116.10:4081 , piece_id=COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.263706679 (502h53m54.134789724s) [noise=true , address=45.91.116.5:4298 , piece_id=UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.265669376 (502h53m54.136752421s) [noise=true , address=45.91.116.14:4190 , piece_id=QVVD5WG72ZBU6RPKGTC7XH4SHE7MHN5CZNRQXEV4LH2D6DANCRYA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.267363312 (502h53m54.138446357s) [noise=true , address=145.79.227.12:4767 , piece_id=V6WPL6JMRPS4D5PNJS7SLRYETDYYLBGNOREQM6BUGDHL37ADRM6Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.268947208 (502h53m54.140030253s) [noise=true , address=145.79.227.35:4482 , piece_id=RU5SDMPVQRHI5RBT4ZHKH3AAXVTWUAZ4ZQHNODEC3LVQJTIEZV4Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.270570268 (502h53m54.141653303s) [noise=true , address=145.79.227.52:4447 , piece_id=PEXVWGYU73F4YTWHREC7UPEYVEED3YE5YFJUUXZKSBRJAL2D62WQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.270782973 (502h53m54.141866008s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:55.983231799 (502h53m55.854314834s) [upload=441327]: Next returned nil segment
Dec 26 14:22:55.983245886 (502h53m55.854328921s) [upload=441327]: Waiting for error group managing segments...
Dec 26 14:22:56.273617707 (502h53m56.144700752s) [noise=true , address=145.79.227.52:4447 , piece_id=PEXVWGYU73F4YTWHREC7UPEYVEED3YE5YFJUUXZKSBRJAL2D62WQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.286562366 (502h53m56.157645401s) [noise=true , address=145.79.227.43:4159 , piece_id=MA4AGRGCXRSJK7IQ2ADMKF7IA5KMN65HOF6PN2DN6ZEH7WREWKLQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.286986444 (502h53m56.158069479s) [noise=true , address=145.79.227.49:4337 , piece_id=JFFJUSVLZK3EOBQA5U43FLHBLFFRPMVCTMTQFJABLLPJVUNP25OA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.287817126 (502h53m56.158900161s) [noise=true , address=145.79.227.19:4078 , piece_id=AMIDTE37DZPBL2NBJW7F4D7ZL4KUCW5QHV66VG3CSXOYRVMY3PAA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.292236150 (502h53m56.163319195s) [noise=true , address=145.79.227.15:4876 , piece_id=CJJRQUTBIKPIHFMOG72UTQL5HGNE5KOT6UIMEE37AWIR23HQOPDA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.293599676 (502h53m56.164682712s) [noise=true , address=145.79.227.12:4767 , piece_id=V6WPL6JMRPS4D5PNJS7SLRYETDYYLBGNOREQM6BUGDHL37ADRM6Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.295499874 (502h53m56.166582909s) [noise=true , address=145.79.227.5:4222 , piece_id=ALTOF6FDW4AJZDSTMGMIVJO74WCAESWPYDNWXOJSJYETXBY6DWSA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.300608212 (502h53m56.171691247s) [noise=true , address=145.79.227.35:4482 , piece_id=RU5SDMPVQRHI5RBT4ZHKH3AAXVTWUAZ4ZQHNODEC3LVQJTIEZV4Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.555432614 (502h53m56.426515679s) [noise=true , address=77.90.31.21:4880 , piece_id=D7XLYLXHWB6LOC2F7ZETQDBGBRWL44KCXCQ4ONXC2PONOQQT6P4Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.610196575 (502h53m56.48127962s) [noise=true , address=45.91.116.10:4081 , piece_id=COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.618211516 (502h53m56.489294551s) [noise=true , address=45.91.116.5:4298 , piece_id=UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.390478915 (502h53m57.26156196s) [noise=true , address=45.91.116.7:4010 , piece_id=VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.519337255 (502h53m57.39042029s) [noise=true , address=45.91.116.14:4190 , piece_id=QVVD5WG72ZBU6RPKGTC7XH4SHE7MHN5CZNRQXEV4LH2D6DANCRYA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.519356741 (502h53m57.390439787s) [seg_pos={0 0}, upload=441327]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:22:57.519535934 (502h53m57.390618969s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection
Dec 26 14:22:57.519542285 (502h53m57.39062532s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519617609 (502h53m57.390700654s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 26 14:22:57.519634510 (502h53m57.390717545s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519892753 (502h53m57.390975788s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection
Dec 26 14:22:57.519902731 (502h53m57.390985766s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519931987 (502h53m57.391015022s) [seg_pos={0 0}, upload=441327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:22:57.520039532 (502h53m57.391122567s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:22:57.520046796 (502h53m57.391129831s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:22:57.810693229 (502h53m57.681776264s) [upload=441327]: Done uploading segments...
Dec 26 14:22:57.810727464 (502h53m57.6818105s) [upload=441327]: Upload finished. err: <nil>
Duration: 5.820879578s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
7.455223667s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:23:27 UTC)
orbiter warning: context deadline exceeded
Duration: 7.455251831s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XXXXX.X..XX..XX...X...X...........X..X.X...........X....XX] duration:
30.003653399s trace:
https://jaeger.dev.storj.io/trace/296cad4c0b00743d
FAIL (2025-12-26 14:23:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ) context deadline exceeded
======LOGS======
ing segments...
Dec 26 14:22:54.002229510 (502h53m53.873312545s) [upload=441327]: Got next segment. Inline: false
Dec 26 14:22:54.002280497 (502h53m53.873363532s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:22:54.002289053 (502h53m53.873372089s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:22:54.256285839 (502h53m54.127368874s) [seg_pos={0 0}, upload=441327]: Begin upload segment...
Dec 26 14:22:54.256338528 (502h53m54.127421574s) [seg_pos={0 0}, upload=441327]: Done begin upload segment.
Dec 26 14:22:54.256358307 (502h53m54.127441352s) [noise=true , address=45.91.116.7:4010 , piece_id=VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.256369878 (502h53m54.127452923s) [noise=true , address=145.79.227.43:4159 , piece_id=MA4AGRGCXRSJK7IQ2ADMKF7IA5KMN65HOF6PN2DN6ZEH7WREWKLQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257039414 (502h53m54.128122449s) [noise=true , address=77.90.31.21:4880 , piece_id=D7XLYLXHWB6LOC2F7ZETQDBGBRWL44KCXCQ4ONXC2PONOQQT6P4Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257520890 (502h53m54.128603935s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.257943105 (502h53m54.12902615s) [noise=true , address=145.79.227.19:4078 , piece_id=AMIDTE37DZPBL2NBJW7F4D7ZL4KUCW5QHV66VG3CSXOYRVMY3PAA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.259271624 (502h53m54.130354669s) [noise=true , address=145.79.227.15:4876 , piece_id=CJJRQUTBIKPIHFMOG72UTQL5HGNE5KOT6UIMEE37AWIR23HQOPDA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.259721591 (502h53m54.130804626s) [noise=true , address=145.79.227.5:4222 , piece_id=ALTOF6FDW4AJZDSTMGMIVJO74WCAESWPYDNWXOJSJYETXBY6DWSA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.260150327 (502h53m54.131233362s) [noise=true , address=145.79.227.49:4337 , piece_id=JFFJUSVLZK3EOBQA5U43FLHBLFFRPMVCTMTQFJABLLPJVUNP25OA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.261389848 (502h53m54.132472893s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.261633412 (502h53m54.132716467s) [noise=true , address=45.91.116.10:4081 , piece_id=COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.263706679 (502h53m54.134789724s) [noise=true , address=45.91.116.5:4298 , piece_id=UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.265669376 (502h53m54.136752421s) [noise=true , address=45.91.116.14:4190 , piece_id=QVVD5WG72ZBU6RPKGTC7XH4SHE7MHN5CZNRQXEV4LH2D6DANCRYA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.267363312 (502h53m54.138446357s) [noise=true , address=145.79.227.12:4767 , piece_id=V6WPL6JMRPS4D5PNJS7SLRYETDYYLBGNOREQM6BUGDHL37ADRM6Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.268947208 (502h53m54.140030253s) [noise=true , address=145.79.227.35:4482 , piece_id=RU5SDMPVQRHI5RBT4ZHKH3AAXVTWUAZ4ZQHNODEC3LVQJTIEZV4Q, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.270570268 (502h53m54.141653303s) [noise=true , address=145.79.227.52:4447 , piece_id=PEXVWGYU73F4YTWHREC7UPEYVEED3YE5YFJUUXZKSBRJAL2D62WQ, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:54.270782973 (502h53m54.141866008s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Uploading piece...
Dec 26 14:22:55.983231799 (502h53m55.854314834s) [upload=441327]: Next returned nil segment
Dec 26 14:22:55.983245886 (502h53m55.854328921s) [upload=441327]: Waiting for error group managing segments...
Dec 26 14:22:56.273617707 (502h53m56.144700752s) [noise=true , address=145.79.227.52:4447 , piece_id=PEXVWGYU73F4YTWHREC7UPEYVEED3YE5YFJUUXZKSBRJAL2D62WQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.286562366 (502h53m56.157645401s) [noise=true , address=145.79.227.43:4159 , piece_id=MA4AGRGCXRSJK7IQ2ADMKF7IA5KMN65HOF6PN2DN6ZEH7WREWKLQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.286986444 (502h53m56.158069479s) [noise=true , address=145.79.227.49:4337 , piece_id=JFFJUSVLZK3EOBQA5U43FLHBLFFRPMVCTMTQFJABLLPJVUNP25OA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.287817126 (502h53m56.158900161s) [noise=true , address=145.79.227.19:4078 , piece_id=AMIDTE37DZPBL2NBJW7F4D7ZL4KUCW5QHV66VG3CSXOYRVMY3PAA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.292236150 (502h53m56.163319195s) [noise=true , address=145.79.227.15:4876 , piece_id=CJJRQUTBIKPIHFMOG72UTQL5HGNE5KOT6UIMEE37AWIR23HQOPDA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.293599676 (502h53m56.164682712s) [noise=true , address=145.79.227.12:4767 , piece_id=V6WPL6JMRPS4D5PNJS7SLRYETDYYLBGNOREQM6BUGDHL37ADRM6Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.295499874 (502h53m56.166582909s) [noise=true , address=145.79.227.5:4222 , piece_id=ALTOF6FDW4AJZDSTMGMIVJO74WCAESWPYDNWXOJSJYETXBY6DWSA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.300608212 (502h53m56.171691247s) [noise=true , address=145.79.227.35:4482 , piece_id=RU5SDMPVQRHI5RBT4ZHKH3AAXVTWUAZ4ZQHNODEC3LVQJTIEZV4Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.555432614 (502h53m56.426515679s) [noise=true , address=77.90.31.21:4880 , piece_id=D7XLYLXHWB6LOC2F7ZETQDBGBRWL44KCXCQ4ONXC2PONOQQT6P4Q, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.610196575 (502h53m56.48127962s) [noise=true , address=45.91.116.10:4081 , piece_id=COZS4SOHDMTG32PTBJACKO6EM6RFHN3CITMS6KS3ENY5BHMVCWQQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:56.618211516 (502h53m56.489294551s) [noise=true , address=45.91.116.5:4298 , piece_id=UUQMDXXJTHDW4J4GEHZAZA42VZATWWHL5Q3KXUJDR5UNR4VXAHXQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.390478915 (502h53m57.26156196s) [noise=true , address=45.91.116.7:4010 , piece_id=VXUQ6RHGYJUV6ICNZ7JBEF4I3TN4BFEEVIW7RHR6XNMME2BC3DUA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.519337255 (502h53m57.39042029s) [noise=true , address=45.91.116.14:4190 , piece_id=QVVD5WG72ZBU6RPKGTC7XH4SHE7MHN5CZNRQXEV4LH2D6DANCRYA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: <nil>
Dec 26 14:22:57.519356741 (502h53m57.390439787s) [seg_pos={0 0}, upload=441327]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:22:57.519535934 (502h53m57.390618969s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34362->77.90.31.18:4808: use of closed network connection
Dec 26 14:22:57.519542285 (502h53m57.39062532s) [noise=true , address=77.90.31.18:4808 , piece_id=VUPAT3EJTP3SLGLLWVC5LI6WEI573EVRIY4ZEUFL7PA34W7WZ2SA, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519617609 (502h53m57.390700654s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 26 14:22:57.519634510 (502h53m57.390717545s) [noise=true , address=45.91.116.13:4156 , piece_id=XT7MOCBM25KYGKEKHMHQNOLPINFNG3Z7KAQED2MXMZDNZRRUVWIQ, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519892753 (502h53m57.390975788s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52864->45.91.116.6:4974: use of closed network connection
Dec 26 14:22:57.519902731 (502h53m57.390985766s) [noise=true , address=45.91.116.6:4974 , piece_id=PNNXP3IATRDSHHLTEYCWCGG33WJ44LWUKYNPHT77SZN76DKHQCXA, seg_pos={0 0}, upload=441327]: Long tail cancellation due to optimal threshold
Dec 26 14:22:57.519931987 (502h53m57.391015022s) [seg_pos={0 0}, upload=441327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:22:57.520039532 (502h53m57.391122567s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:22:57.520046796 (502h53m57.391129831s) [upload=441327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:22:57.810693229 (502h53m57.681776264s) [upload=441327]: Done uploading segments...
Dec 26 14:22:57.810727464 (502h53m57.6818105s) [upload=441327]: Upload finished. err: <nil>
Duration: 30.003675561s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXXX] duration:
5.00105301s trace:
https://jaeger.dev.storj.io/trace/2163fafe946ce485
FAIL (2025-12-26 14:23:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001078639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
12.339060767s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:23:08 UTC)
orbiter warning: context deadline exceeded
Duration: 12.339091186s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXXX] duration:
5.000989367s trace:
https://jaeger.dev.storj.io/trace/2eda344e4b1fbda8
FAIL (2025-12-26 14:23:01 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00101681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XXX] duration:
5.000755202s trace:
https://jaeger.dev.storj.io/trace/71c59991a0e2f9a0
FAIL (2025-12-26 14:22:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000839493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
18.785930727s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:22:39 UTC)
orbiter warning: context deadline exceeded
Duration: 18.785944943s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X.X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.XX] duration:
1m0.003019377s trace:
https://jaeger.dev.storj.io/trace/5628ee224f269903
FAIL (2025-12-26 14:22:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
74 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857678324 (502h52m34.728761359s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.857725413 (502h52m34.728808448s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858089236 (502h52m34.729172282s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858285019 (502h52m34.729368094s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858517993 (502h52m34.729601038s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858634183 (502h52m34.729717219s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858879741 (502h52m34.729962786s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.858927523 (502h52m34.730010558s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859231562 (502h52m34.730314597s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859259344 (502h52m34.730342379s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859549567 (502h52m34.730632602s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859614772 (502h52m34.730697807s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.859823088 (502h52m34.730906123s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:34.860012338 (502h52m34.731095373s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Uploading piece...
Dec 26 14:21:36.336089476 (502h52m36.207172511s) [upload=441307]: Next returned nil segment
Dec 26 14:21:36.336104385 (502h52m36.20718743s) [upload=441307]: Waiting for error group managing segments...
Dec 26 14:21:36.750189749 (502h52m36.621272794s) [noise=true , address=145.79.227.34:4918 , piece_id=J62W7FCW7JXREODLLYXUFCCVANIXOHLWFSU2CW5RKDHVMMLBPLCQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.768347334 (502h52m36.639430379s) [noise=true , address=145.79.227.47:4281 , piece_id=CXHHX52LXNGJWF4EPP5ZSOXECDBSGLA3LILKFOGI6FBII5YFEG5A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.773778426 (502h52m36.644861471s) [noise=true , address=145.79.227.52:4465 , piece_id=GUNZ23PIXZKPXRZOW3I6JPL5HQ6IWZLQDW3ILZ2QM7YEMUPCQQ6A, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.787047572 (502h52m36.658130617s) [noise=true , address=145.79.227.52:4469 , piece_id=XBFRMOIYGHD72YQRE5MOGRYYOFBDLOBB3RM7PS4GYDYBDPNKTS5Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.921335322 (502h52m36.792418367s) [noise=true , address=77.90.31.16:4741 , piece_id=HJFGF6P4BLXNKBM25EWO623276OCTXQ6H5U2ILVQIOYSVQTW77FA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.941684710 (502h52m36.812767755s) [noise=true , address=77.90.31.15:4709 , piece_id=B7G4ROEUDS433F7EXMDM36WVS2QO4JHP5BD5EX5MUH2I5VXNIR7Q, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.955124502 (502h52m36.826207547s) [noise=true , address=77.90.31.19:4866 , piece_id=LZMM5DKIQ6M7VAV5WLSEIHTYZXXT5BPWVIRCUFZAYF3I6XOEUKFA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.965828374 (502h52m36.836911409s) [noise=true , address=77.90.31.18:4809 , piece_id=NIGGUIBWDIEU7D6AKHP4AJE2DYJCVSYUPMSHQUKP3V4XRJK5SXGA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:36.996276633 (502h52m36.867359668s) [noise=true , address=77.90.31.14:4688 , piece_id=YK3EWDMN5ICZHNGVPD5RTJYFXD53IHSC2V4KHUFVP7QTSSKFBLWQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.003628292 (502h52m36.874711367s) [noise=true , address=77.90.31.11:4574 , piece_id=3YXHRWTOF6MEFUMFNWCGEZE2G6KMGR76RVPLKO5AHJFUSOMV7CVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.451102214 (502h52m37.322185249s) [noise=true , address=45.91.116.14:4208 , piece_id=OH52WYKUM4O4YNUBRKPKHSQRMLSZIJ2Z36XNECTKWBUTVVHINAGQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.616340388 (502h52m37.487423433s) [noise=true , address=145.79.227.30:4793 , piece_id=5QNELH5BMMH37SQXYSUSZKHAEGRQLJC3OHHOPBP64MY2AFEIIKSQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.689866315 (502h52m37.56094935s) [noise=true , address=145.79.227.44:4210 , piece_id=2QC3GNJQ53M3NHEWMEDGVINQLJ6F6WT2QZ3ZQQNA5HGBNMB2EEAA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720684999 (502h52m37.591768044s) [noise=true , address=145.79.227.40:4078 , piece_id=AWQV3PHI6C3L7JYYIS4FOX4753DOZE2FRLZQ4HNXDQIYRE6RSFAQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: <nil>
Dec 26 14:21:37.720704766 (502h52m37.591787811s) [seg_pos={0 0}, upload=441307]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:21:37.720881473 (502h52m37.591964518s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF): protocol: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52018->77.90.31.12:4613: use of closed network connection
Dec 26 14:21:37.720889007 (502h52m37.591972042s) [noise=true , address=77.90.31.12:4613 , piece_id=FAYIKHKHSJVGWE4LVPEYRBVVT4A5W5ZMGSGXRCMIRKD4X7TE5OVQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.720987195 (502h52m37.59207024s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Eki7jYegwvkdixR5pznQBbSeJPn9GjGofxnUApidQF4kRNBK5): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721006190 (502h52m37.592089235s) [noise=true , address=45.91.116.4:4963 , piece_id=Q7BGBKV2EMVRXRKDXOKXXB6JVVQ3V5ITH5Q6AVYHY7IO4DF3QTZQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721293317 (502h52m37.592376362s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:37.721305220 (502h52m37.592388255s) [noise=true , address=45.91.116.12:4238 , piece_id=J2B2PDJK5I4KBMOB7DQZ2VFMLK7C2KEFQEEUIET54W7HPGQ5ALLQ, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721422834 (502h52m37.592505869s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34140->64.49.13.169:4709: use of closed network connection
Dec 26 14:21:37.721432902 (502h52m37.592515938s) [noise=true , address=64.49.13.169:4709 , piece_id=WDN3RENLR24KDEBLRBITZK5NCLAEFGQV3LH6PLFKQ4IPL2CJDHKA, seg_pos={0 0}, upload=441307]: Long tail cancellation due to optimal threshold
Dec 26 14:21:37.721452681 (502h52m37.592535716s) [seg_pos={0 0}, upload=441307]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:21:37.721520750 (502h52m37.592603795s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:37.721525659 (502h52m37.592608694s) [upload=441307]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:38.022724778 (502h52m37.893807813s) [upload=441307]: Done uploading segments...
Dec 26 14:21:38.022745177 (502h52m37.893828222s) [upload=441307]: Upload finished. err: <nil>
Duration: 1m0.003032162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XXXXX.X..XX..XX...X...X...........X..X.X...........X....X] duration:
30.003543016s trace:
https://jaeger.dev.storj.io/trace/4171751b9956339a
FAIL (2025-12-26 14:22:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:21:50.385127592 (502h52m50.256210637s) [upload=441312]: Starting upload
Dec 26 14:21:50.385253572 (502h52m50.256336617s) [upload=441312]: Done starting upload
Dec 26 14:21:50.385368611 (502h52m50.256451656s) [upload=441312]: Uploading segments...
Dec 26 14:21:50.385536150 (502h52m50.256619195s) [upload=441312]: Got next segment. Inline: false
Dec 26 14:21:50.385581737 (502h52m50.256664792s) [upload=441312]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:21:50.385587618 (502h52m50.256670663s) [upload=441312]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:21:50.629686607 (502h52m50.500769652s) [seg_pos={0 0}, upload=441312]: Begin upload segment...
Dec 26 14:21:50.629745730 (502h52m50.500828765s) [seg_pos={0 0}, upload=441312]: Done begin upload segment.
Dec 26 14:21:50.629773744 (502h52m50.500856789s) [noise=true , address=145.79.227.22:4511 , piece_id=73O3NTCQSASJL47VGQMTUXUVQN6ENSRHER3TGBP3SSW4AQ2WQVRA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.629782180 (502h52m50.500865255s) [noise=true , address=77.90.31.15:4700 , piece_id=ZJQUZJQVMXEOLHKQ4CRIG5U47VVMTGVWWS7X27UHARTOEMRPBOVA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.630288765 (502h52m50.5013718s) [noise=true , address=45.91.116.12:4226 , piece_id=QYBQEVEWGW2RXR6F4LNEYXOQYIEAB2J2RA2R5V3ORVKNOK35YS6A, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.630642618 (502h52m50.501725653s) [noise=true , address=145.79.227.45:4231 , piece_id=IDEKYX2YLDULZXJ5BD7LWPW4FIBXBICA44N25SWVNGCXHG5OQQ3A, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.630944673 (502h52m50.502027708s) [noise=true , address=145.79.227.46:4603 , piece_id=QFZ5YAK6RVZIL5R2HKCRISTDXLDK6YGDV3VV25WVWA3QUTDQYNYA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.631275613 (502h52m50.502358658s) [noise=true , address=145.79.227.52:4445 , piece_id=5FL5YNFO6RIS46HKJJ7SDM3L46EXK6MI5BFVPETOUNDJP3FQQF3A, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.631644786 (502h52m50.502727821s) [noise=true , address=45.91.116.9:4261 , piece_id=XXG5BIQAG2SUGQIWGOQZQBAGO5FGUNCOOUQB2V4LMZKDCWBMKGKQ, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.632083271 (502h52m50.503166316s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAER6JS2F7UBUZNVIIJHCJ5NSM3Q4GCWBYUTD3KWDM6P3CHD4SUQ, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.632522448 (502h52m50.503605483s) [noise=true , address=45.91.116.14:4192 , piece_id=L45POZFL2GQHSYYB2WXDKDDUJGUSQGCY7E5QDFCXGFNLFHCW6SHA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.634268743 (502h52m50.505351778s) [noise=true , address=145.79.227.43:4159 , piece_id=EOR23HUO3ZXCGN2OATPEPHW6LJZIDNABTUAA7RFSQU55KPYSVK2Q, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.634368122 (502h52m50.505451157s) [noise=true , address=145.79.227.5:4223 , piece_id=EXY444KD6MNUAM3IS4PYHV44HZRX5TA4QQK3XSOFJNEXVOEPJCRA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.635967758 (502h52m50.507050803s) [noise=true , address=145.79.227.14:4840 , piece_id=WR2HZVP5HBQC3V3PSGM3DEVXYBPRUXQ73BEW44POZ4MS24CGH6OQ, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.636137702 (502h52m50.507220737s) [noise=true , address=145.79.227.6:4257 , piece_id=EIZIIT3FUZBMTJORGEOG7AXIML4TLUUABV5ITASDSVMQGDUY25AA, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.637625715 (502h52m50.50870875s) [noise=true , address=145.79.227.39:4013 , piece_id=AJPWKKDVZI5L437GYFZEKDEZ6ZGEBU6FSWZP6TY7XZ3ZXJXPMN4A, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.638301773 (502h52m50.509384818s) [noise=true , address=45.91.116.4:4937 , piece_id=S7CUSE4SVFFVKOPEWKIFNLTBJWN6ELDYNMMOV4HRD7J5KQZSRSOQ, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:50.641554746 (502h52m50.512637781s) [noise=true , address=145.79.227.29:4943 , piece_id=YBTHOUM3I5HRIP3Q2FOJJ2NJX43ZK6K3PQSEXIS5PXRELTLLFX3Q, seg_pos={0 0}, upload=441312]: Uploading piece...
Dec 26 14:21:52.264409008 (502h52m52.135492043s) [upload=441312]: Next returned nil segment
Dec 26 14:21:52.264421311 (502h52m52.135504356s) [upload=441312]: Waiting for error group managing segments...
Dec 26 14:21:52.648180659 (502h52m52.519263704s) [noise=true , address=145.79.227.14:4840 , piece_id=WR2HZVP5HBQC3V3PSGM3DEVXYBPRUXQ73BEW44POZ4MS24CGH6OQ, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.649754796 (502h52m52.520837831s) [noise=true , address=145.79.227.6:4257 , piece_id=EIZIIT3FUZBMTJORGEOG7AXIML4TLUUABV5ITASDSVMQGDUY25AA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.656996645 (502h52m52.528079691s) [noise=true , address=145.79.227.5:4223 , piece_id=EXY444KD6MNUAM3IS4PYHV44HZRX5TA4QQK3XSOFJNEXVOEPJCRA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.658695871 (502h52m52.529778906s) [noise=true , address=145.79.227.45:4231 , piece_id=IDEKYX2YLDULZXJ5BD7LWPW4FIBXBICA44N25SWVNGCXHG5OQQ3A, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.663705481 (502h52m52.534788526s) [noise=true , address=145.79.227.29:4943 , piece_id=YBTHOUM3I5HRIP3Q2FOJJ2NJX43ZK6K3PQSEXIS5PXRELTLLFX3Q, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.663909569 (502h52m52.534992614s) [noise=true , address=145.79.227.22:4511 , piece_id=73O3NTCQSASJL47VGQMTUXUVQN6ENSRHER3TGBP3SSW4AQ2WQVRA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.672330925 (502h52m52.54341397s) [noise=true , address=145.79.227.39:4013 , piece_id=AJPWKKDVZI5L437GYFZEKDEZ6ZGEBU6FSWZP6TY7XZ3ZXJXPMN4A, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.673447161 (502h52m52.544530196s) [noise=true , address=145.79.227.46:4603 , piece_id=QFZ5YAK6RVZIL5R2HKCRISTDXLDK6YGDV3VV25WVWA3QUTDQYNYA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.678328997 (502h52m52.549412032s) [noise=true , address=145.79.227.52:4445 , piece_id=5FL5YNFO6RIS46HKJJ7SDM3L46EXK6MI5BFVPETOUNDJP3FQQF3A, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:52.924375825 (502h52m52.79545886s) [noise=true , address=45.91.116.4:4937 , piece_id=S7CUSE4SVFFVKOPEWKIFNLTBJWN6ELDYNMMOV4HRD7J5KQZSRSOQ, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:53.014898657 (502h52m52.885981702s) [noise=true , address=45.91.116.14:4192 , piece_id=L45POZFL2GQHSYYB2WXDKDDUJGUSQGCY7E5QDFCXGFNLFHCW6SHA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:53.017638784 (502h52m52.888721829s) [noise=true , address=45.91.116.9:4261 , piece_id=XXG5BIQAG2SUGQIWGOQZQBAGO5FGUNCOOUQB2V4LMZKDCWBMKGKQ, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:53.860567585 (502h52m53.73165062s) [noise=true , address=77.90.31.10:4519 , piece_id=ZAER6JS2F7UBUZNVIIJHCJ5NSM3Q4GCWBYUTD3KWDM6P3CHD4SUQ, seg_pos={0 0}, upload=441312]: Done uploading piece. err: <nil>
Dec 26 14:21:53.860592252 (502h52m53.731675297s) [seg_pos={0 0}, upload=441312]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:21:53.860829584 (502h52m53.731912619s) [noise=true , address=145.79.227.43:4159 , piece_id=EOR23HUO3ZXCGN2OATPEPHW6LJZIDNABTUAA7RFSQU55KPYSVK2Q, seg_pos={0 0}, upload=441312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:53.860839342 (502h52m53.731922387s) [noise=true , address=145.79.227.43:4159 , piece_id=EOR23HUO3ZXCGN2OATPEPHW6LJZIDNABTUAA7RFSQU55KPYSVK2Q, seg_pos={0 0}, upload=441312]: Long tail cancellation due to optimal threshold
Dec 26 14:21:53.861026128 (502h52m53.732109173s) [noise=true , address=45.91.116.12:4226 , piece_id=QYBQEVEWGW2RXR6F4LNEYXOQYIEAB2J2RA2R5V3ORVKNOK35YS6A, seg_pos={0 0}, upload=441312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:53.861044713 (502h52m53.732127769s) [noise=true , address=45.91.116.12:4226 , piece_id=QYBQEVEWGW2RXR6F4LNEYXOQYIEAB2J2RA2R5V3ORVKNOK35YS6A, seg_pos={0 0}, upload=441312]: Long tail cancellation due to optimal threshold
Dec 26 14:21:53.861193557 (502h52m53.732276602s) [noise=true , address=77.90.31.15:4700 , piece_id=ZJQUZJQVMXEOLHKQ4CRIG5U47VVMTGVWWS7X27UHARTOEMRPBOVA, seg_pos={0 0}, upload=441312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 26 14:21:53.861204237 (502h52m53.732287272s) [noise=true , address=77.90.31.15:4700 , piece_id=ZJQUZJQVMXEOLHKQ4CRIG5U47VVMTGVWWS7X27UHARTOEMRPBOVA, seg_pos={0 0}, upload=441312]: Long tail cancellation due to optimal threshold
Dec 26 14:21:53.861226660 (502h52m53.732309695s) [seg_pos={0 0}, upload=441312]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:21:53.861312182 (502h52m53.732395227s) [upload=441312]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:21:53.861319597 (502h52m53.732402642s) [upload=441312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:21:54.147941417 (502h52m54.019024462s) [upload=441312]: Done uploading segments...
Dec 26 14:21:54.147962617 (502h52m54.019045652s) [upload=441312]: Upload finished. err: <nil>
Duration: 30.003610695s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.XX] duration:
5.00430991s trace:
https://jaeger.dev.storj.io/trace/1c01886eb3a5f4b1
FAIL (2025-12-26 14:22:03 UTC)
orbiter warning: context deadline exceeded
Duration: 5.004338013s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
11.540563861s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:21:57 UTC)
orbiter warning: context deadline exceeded
Duration: 11.540591373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
19.106640703s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:21:54 UTC)
orbiter warning: context deadline exceeded
Duration: 19.106670429s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.XXXXX.X..XX..XX...X...X...........X..X.X...........X....] duration:
5.836487864s trace:
https://jaeger.dev.storj.io/trace/3e839c16aad153a6
FAIL (2025-12-26 14:21:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q) context deadline exceeded
======LOGS======
14:18:47.356998944 (502h49m47.228081979s) [upload=441269]: Starting upload
Dec 26 14:18:47.357084037 (502h49m47.228167082s) [upload=441269]: Done starting upload
Dec 26 14:18:47.357123442 (502h49m47.228206487s) [upload=441269]: Uploading segments...
Dec 26 14:18:47.357439023 (502h49m47.228522068s) [upload=441269]: Got next segment. Inline: false
Dec 26 14:18:47.357479720 (502h49m47.228562755s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:18:47.357484669 (502h49m47.228567704s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:18:47.606177667 (502h49m47.477260702s) [seg_pos={0 0}, upload=441269]: Begin upload segment...
Dec 26 14:18:47.606230587 (502h49m47.477313622s) [seg_pos={0 0}, upload=441269]: Done begin upload segment.
Dec 26 14:18:47.606292986 (502h49m47.477376031s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606313534 (502h49m47.477396579s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606988650 (502h49m47.478071686s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607483903 (502h49m47.478566938s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607791419 (502h49m47.478874464s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608176933 (502h49m47.479259968s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608477235 (502h49m47.47956028s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608795461 (502h49m47.479878496s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609114679 (502h49m47.480197724s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609584413 (502h49m47.480667448s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610006628 (502h49m47.481089663s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610139139 (502h49m47.481222174s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610385598 (502h49m47.481468633s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610784729 (502h49m47.481867764s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611152939 (502h49m47.482235974s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611535077 (502h49m47.482618112s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:49.321525799 (502h49m49.192608844s) [upload=441269]: Next returned nil segment
Dec 26 14:18:49.321539244 (502h49m49.192622279s) [upload=441269]: Waiting for error group managing segments...
Dec 26 14:18:49.600328349 (502h49m49.471411384s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.607345371 (502h49m49.478428406s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.615918155 (502h49m49.48700122s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.620998108 (502h49m49.492081153s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.621590276 (502h49m49.492673311s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622164068 (502h49m49.493247103s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622514555 (502h49m49.4935976s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.626668937 (502h49m49.497751982s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.638758748 (502h49m49.509841793s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.641537970 (502h49m49.512621015s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.654182768 (502h49m49.525265803s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.900555837 (502h49m49.771638882s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919343812 (502h49m49.790426857s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919368009 (502h49m49.790451054s) [seg_pos={0 0}, upload=441269]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:18:49.919595642 (502h49m49.790678697s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection
Dec 26 14:18:49.919604949 (502h49m49.790688004s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919779662 (502h49m49.790862707s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919795673 (502h49m49.790878718s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919928807 (502h49m49.791011852s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919934828 (502h49m49.791017863s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919959485 (502h49m49.79104252s) [seg_pos={0 0}, upload=441269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:18:49.920045398 (502h49m49.791128443s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:18:49.920050468 (502h49m49.791133503s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:18:50.214541632 (502h49m50.085624687s) [upload=441269]: Done uploading segments...
Dec 26 14:18:50.214565568 (502h49m50.085648613s) [upload=441269]: Upload finished. err: <nil>
Duration: 5.836530165s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
11.499270055s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:21:39 UTC)
orbiter warning: context deadline exceeded
Duration: 11.499298048s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.X] duration:
5.00154231s trace:
https://jaeger.dev.storj.io/trace/4dd0c0c09dce556f
FAIL (2025-12-26 14:21:36 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001566174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.X.X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.X] duration:
1m0.003324581s trace:
https://jaeger.dev.storj.io/trace/69db290bb62f6f6f
FAIL (2025-12-26 14:21:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4:20:34.612093174 (502h51m34.483176209s) [noise=true , address=45.91.116.11:4135 , piece_id=KJTZ4JSZDPHHBZA5H76DH66ADYKZNOXWHIRJA5DZPAEIVUQCO3AA, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.612597314 (502h51m34.48368035s) [noise=true , address=145.79.227.22:4516 , piece_id=4WCCQX6HF2CWKKKRJCBZBZFKAUKM5DUJ4HXFDF5APRBF37UC3FFQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.613001494 (502h51m34.48408453s) [noise=true , address=145.79.227.52:4465 , piece_id=P3AOILEDA4DAYJ3YL5FBCZNNTCY55LJDT5AIR75VEK6EZ4EJKTGA, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.613411354 (502h51m34.484494389s) [noise=true , address=145.79.227.44:4208 , piece_id=AVTOUGHJGCHSYWON3COIXE5ZU67OGJKD5BX2D4BQW5LSEDOAP7XA, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.613844189 (502h51m34.484927234s) [noise=true , address=77.90.31.11:4562 , piece_id=3SSZGFHFKLKS2OW75TQOKCQ5MW2WQ626KZ2WD4IVH5VUG6WHV7VA, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614098252 (502h51m34.485181287s) [noise=true , address=77.90.31.18:4824 , piece_id=VH6GAPOIUJ3543NUEWFQILXPXEKREA33TRIQJYKE5KYJWNWPWECQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614272564 (502h51m34.485355599s) [noise=true , address=145.79.227.43:4180 , piece_id=FYILNPFHMRZBREI6OLGR4QWJBJAZ2UMDWY37J3R3QSQMXAFA7ISQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614580310 (502h51m34.485663346s) [noise=true , address=77.90.31.12:4598 , piece_id=7HJ64CSUKB3JEX7WU5R64KJIPJ6U22GEAXJ4JNLNLQ7PX7TIQRHQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614627370 (502h51m34.485710415s) [noise=true , address=145.79.227.34:4920 , piece_id=77SZLOJ7C3DQK5L3FJAE52XO7IP4LS57FB463S66VAJW64OPZW5A, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614985362 (502h51m34.486068397s) [noise=true , address=193.149.164.143:4699 , piece_id=RNPEGJ7Y2HFZHYTC7FILSVJ5DLAJHRJNFNNLKHKH66DW2HSFDI2Q, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.614993978 (502h51m34.486107532s) [noise=true , address=145.79.227.47:4279 , piece_id=YKWFDIEBUDTV5OFBHBZSAQHVZ54GLFPRIC2LO7LHAGLAFASRWYTQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.615391775 (502h51m34.48647481s) [noise=true , address=77.90.31.19:4853 , piece_id=LVIPSDE5AJSKL5GHGFYEEWDSMCIX6WGL3WZ2AEHLWP2OFGUHWFVQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.615415000 (502h51m34.486498035s) [noise=true , address=145.79.227.30:4767 , piece_id=D5XPTAVZZBTB5GCV76PLGF4JMIEB4FGRKPJTNOQIHLD226L34GFQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.615836272 (502h51m34.486919307s) [noise=true , address=77.90.31.14:4696 , piece_id=7ETRNKKC7MXGH5WIWAS7AA3UM2COFGE2F3WHOQ5Z23ITNHUHDHMQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.616399053 (502h51m34.487482099s) [noise=true , address=77.90.31.16:4760 , piece_id=ORABPDE2TEIWCFZQUXK7JN3O7VAX5HFZJ3VE5E3VEHFLY56ARLWA, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.616888065 (502h51m34.4879711s) [noise=true , address=77.90.31.15:4719 , piece_id=U67EXB5OYTP75X3GGP2WIQNGRJKTS3EUKXOWDHIUUNHPRLJVATVQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:34.617527012 (502h51m34.488610047s) [noise=true , address=77.90.31.13:4648 , piece_id=EGETJZNXDEUMO4BS2ZCQTX7QEDM2X2KKZEVBBQE7LEHU33SWQFMQ, seg_pos={0 0}, upload=441295]: Uploading piece...
Dec 26 14:20:36.094491008 (502h51m35.965574043s) [upload=441295]: Next returned nil segment
Dec 26 14:20:36.094504754 (502h51m35.965587799s) [upload=441295]: Waiting for error group managing segments...
Dec 26 14:20:36.491852095 (502h51m36.36293514s) [noise=true , address=145.79.227.47:4279 , piece_id=YKWFDIEBUDTV5OFBHBZSAQHVZ54GLFPRIC2LO7LHAGLAFASRWYTQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.511906353 (502h51m36.382989388s) [noise=true , address=145.79.227.22:4516 , piece_id=4WCCQX6HF2CWKKKRJCBZBZFKAUKM5DUJ4HXFDF5APRBF37UC3FFQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.514432512 (502h51m36.385515547s) [noise=true , address=145.79.227.52:4465 , piece_id=P3AOILEDA4DAYJ3YL5FBCZNNTCY55LJDT5AIR75VEK6EZ4EJKTGA, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.811093990 (502h51m36.682177035s) [noise=true , address=45.91.116.11:4135 , piece_id=KJTZ4JSZDPHHBZA5H76DH66ADYKZNOXWHIRJA5DZPAEIVUQCO3AA, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.854708800 (502h51m36.725791855s) [noise=true , address=77.90.31.14:4696 , piece_id=7ETRNKKC7MXGH5WIWAS7AA3UM2COFGE2F3WHOQ5Z23ITNHUHDHMQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.869132735 (502h51m36.740215801s) [noise=true , address=77.90.31.19:4853 , piece_id=LVIPSDE5AJSKL5GHGFYEEWDSMCIX6WGL3WZ2AEHLWP2OFGUHWFVQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:36.880436560 (502h51m36.751519595s) [noise=true , address=77.90.31.15:4719 , piece_id=U67EXB5OYTP75X3GGP2WIQNGRJKTS3EUKXOWDHIUUNHPRLJVATVQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.216916259 (502h51m37.087999294s) [noise=true , address=193.149.164.143:4699 , piece_id=RNPEGJ7Y2HFZHYTC7FILSVJ5DLAJHRJNFNNLKHKH66DW2HSFDI2Q, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.366766711 (502h51m37.237849746s) [noise=true , address=77.90.31.18:4824 , piece_id=VH6GAPOIUJ3543NUEWFQILXPXEKREA33TRIQJYKE5KYJWNWPWECQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.376083722 (502h51m37.247166757s) [noise=true , address=145.79.227.30:4767 , piece_id=D5XPTAVZZBTB5GCV76PLGF4JMIEB4FGRKPJTNOQIHLD226L34GFQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.376205905 (502h51m37.24728894s) [noise=true , address=145.79.227.40:4081 , piece_id=GDSK4TRKX6RMTVVRXNBZQNGRXUAN37VG7UCAWTIXHIAX6XVTRF6A, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.482130303 (502h51m37.353213338s) [noise=true , address=145.79.227.44:4208 , piece_id=AVTOUGHJGCHSYWON3COIXE5ZU67OGJKD5BX2D4BQW5LSEDOAP7XA, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.715832194 (502h51m37.58691524s) [noise=true , address=145.79.227.34:4920 , piece_id=77SZLOJ7C3DQK5L3FJAE52XO7IP4LS57FB463S66VAJW64OPZW5A, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.723606779 (502h51m37.594689814s) [noise=true , address=77.90.31.13:4648 , piece_id=EGETJZNXDEUMO4BS2ZCQTX7QEDM2X2KKZEVBBQE7LEHU33SWQFMQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: <nil>
Dec 26 14:20:37.723631526 (502h51m37.594714571s) [seg_pos={0 0}, upload=441295]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:20:37.723912542 (502h51m37.594995597s) [noise=true , address=77.90.31.11:4562 , piece_id=3SSZGFHFKLKS2OW75TQOKCQ5MW2WQ626KZ2WD4IVH5VUG6WHV7VA, seg_pos={0 0}, upload=441295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF
Dec 26 14:20:37.723922080 (502h51m37.595005125s) [noise=true , address=77.90.31.11:4562 , piece_id=3SSZGFHFKLKS2OW75TQOKCQ5MW2WQ626KZ2WD4IVH5VUG6WHV7VA, seg_pos={0 0}, upload=441295]: Long tail cancellation due to optimal threshold
Dec 26 14:20:37.724043210 (502h51m37.595126255s) [noise=true , address=77.90.31.12:4598 , piece_id=7HJ64CSUKB3JEX7WU5R64KJIPJ6U22GEAXJ4JNLNLQ7PX7TIQRHQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Li7UVDCVmainU4mkxtFyGky2wi6989Qu12BCQGgNn6roNX2A3): protocol: expected piece hash; context canceled; EOF
Dec 26 14:20:37.724054111 (502h51m37.595137146s) [noise=true , address=77.90.31.12:4598 , piece_id=7HJ64CSUKB3JEX7WU5R64KJIPJ6U22GEAXJ4JNLNLQ7PX7TIQRHQ, seg_pos={0 0}, upload=441295]: Long tail cancellation due to optimal threshold
Dec 26 14:20:37.724227722 (502h51m37.595310757s) [noise=true , address=145.79.227.43:4180 , piece_id=FYILNPFHMRZBREI6OLGR4QWJBJAZ2UMDWY37J3R3QSQMXAFA7ISQ, seg_pos={0 0}, upload=441295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 26 14:20:37.724236619 (502h51m37.595319664s) [noise=true , address=145.79.227.43:4180 , piece_id=FYILNPFHMRZBREI6OLGR4QWJBJAZ2UMDWY37J3R3QSQMXAFA7ISQ, seg_pos={0 0}, upload=441295]: Long tail cancellation due to optimal threshold
Dec 26 14:20:37.724370293 (502h51m37.595453338s) [noise=true , address=77.90.31.16:4760 , piece_id=ORABPDE2TEIWCFZQUXK7JN3O7VAX5HFZJ3VE5E3VEHFLY56ARLWA, seg_pos={0 0}, upload=441295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1e6p1TD5CqUyWdZjkrxh2kRqMuyDiMUr9eimfeayDAKwhtBrZ8): protocol: expected piece hash; context canceled; EOF
Dec 26 14:20:37.724379811 (502h51m37.595462856s) [noise=true , address=77.90.31.16:4760 , piece_id=ORABPDE2TEIWCFZQUXK7JN3O7VAX5HFZJ3VE5E3VEHFLY56ARLWA, seg_pos={0 0}, upload=441295]: Long tail cancellation due to optimal threshold
Dec 26 14:20:37.724400831 (502h51m37.595483877s) [seg_pos={0 0}, upload=441295]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:20:37.724540588 (502h51m37.595623633s) [upload=441295]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:20:37.724548713 (502h51m37.595631748s) [upload=441295]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:20:38.025023179 (502h51m37.896106235s) [upload=441295]: Done uploading segments...
Dec 26 14:20:38.025049650 (502h51m37.896132695s) [upload=441295]: Upload finished. err: <nil>
Duration: 1m0.003341765s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
10.007939956s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:21:03 UTC)
orbiter warning: context deadline exceeded
Duration: 10.007969442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X..X.XXXXX.X..XX..XX...X...X...........X..X.X...........X...] duration:
5.977590595s trace:
https://jaeger.dev.storj.io/trace/3e839c16aad153a6
FAIL (2025-12-26 14:21:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q) context deadline exceeded
======LOGS======
14:18:47.356998944 (502h49m47.228081979s) [upload=441269]: Starting upload
Dec 26 14:18:47.357084037 (502h49m47.228167082s) [upload=441269]: Done starting upload
Dec 26 14:18:47.357123442 (502h49m47.228206487s) [upload=441269]: Uploading segments...
Dec 26 14:18:47.357439023 (502h49m47.228522068s) [upload=441269]: Got next segment. Inline: false
Dec 26 14:18:47.357479720 (502h49m47.228562755s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:18:47.357484669 (502h49m47.228567704s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:18:47.606177667 (502h49m47.477260702s) [seg_pos={0 0}, upload=441269]: Begin upload segment...
Dec 26 14:18:47.606230587 (502h49m47.477313622s) [seg_pos={0 0}, upload=441269]: Done begin upload segment.
Dec 26 14:18:47.606292986 (502h49m47.477376031s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606313534 (502h49m47.477396579s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606988650 (502h49m47.478071686s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607483903 (502h49m47.478566938s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607791419 (502h49m47.478874464s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608176933 (502h49m47.479259968s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608477235 (502h49m47.47956028s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608795461 (502h49m47.479878496s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609114679 (502h49m47.480197724s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609584413 (502h49m47.480667448s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610006628 (502h49m47.481089663s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610139139 (502h49m47.481222174s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610385598 (502h49m47.481468633s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610784729 (502h49m47.481867764s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611152939 (502h49m47.482235974s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611535077 (502h49m47.482618112s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:49.321525799 (502h49m49.192608844s) [upload=441269]: Next returned nil segment
Dec 26 14:18:49.321539244 (502h49m49.192622279s) [upload=441269]: Waiting for error group managing segments...
Dec 26 14:18:49.600328349 (502h49m49.471411384s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.607345371 (502h49m49.478428406s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.615918155 (502h49m49.48700122s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.620998108 (502h49m49.492081153s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.621590276 (502h49m49.492673311s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622164068 (502h49m49.493247103s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622514555 (502h49m49.4935976s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.626668937 (502h49m49.497751982s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.638758748 (502h49m49.509841793s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.641537970 (502h49m49.512621015s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.654182768 (502h49m49.525265803s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.900555837 (502h49m49.771638882s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919343812 (502h49m49.790426857s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919368009 (502h49m49.790451054s) [seg_pos={0 0}, upload=441269]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:18:49.919595642 (502h49m49.790678697s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection
Dec 26 14:18:49.919604949 (502h49m49.790688004s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919779662 (502h49m49.790862707s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919795673 (502h49m49.790878718s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919928807 (502h49m49.791011852s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919934828 (502h49m49.791017863s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919959485 (502h49m49.79104252s) [seg_pos={0 0}, upload=441269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:18:49.920045398 (502h49m49.791128443s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:18:49.920050468 (502h49m49.791133503s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:18:50.214541632 (502h49m50.085624687s) [upload=441269]: Done uploading segments...
Dec 26 14:18:50.214565568 (502h49m50.085648613s) [upload=441269]: Upload finished. err: <nil>
Duration: 5.977616424s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX.XXXXX.] duration:
4.837577582s trace:
https://jaeger.dev.storj.io/trace/2b580cad51c3ede5
FAIL (2025-12-26 14:20:53 UTC)
orbiter warning: context deadline exceeded
Duration: 4.837606967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
12.939686032s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:20:52 UTC)
orbiter warning: context deadline exceeded
Duration: 12.939700999s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X.....XXX.XX.] duration:
50.862420199s trace:
https://jaeger.dev.storj.io/trace/3e1b037351c3a2e
FAIL (2025-12-26 14:20:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, Piece ID: DGJ7OCZTT447YPGGJ4GAQFIUCJ2DGXGRUSOG4FH2WHJ34SVAZRQA) context deadline exceeded; piecestore: (Node ID: 1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq, Piece ID: I5ETRK2KBL6BQBFQVQ2T3ZRBPPOEILP45ZR2QU2UJ4U5H5XKUHYQ) context deadline exceeded; piecestore: (Node ID: 1TrcoCHhAJeEykP9hJNpygoPSdgFV5v6GzL48JpyNPiAj5ZJRS, Piece ID: H6BEM3BWWOHLBWAOKA4Q72THPYOFZECENPNNKZSAXL3TW4TDU6HQ) context deadline exceeded; piecestore: (Node ID: 12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, Piece ID: K3WQHFZWAZBM3FL7MKBJJEOGIVRXSFZEQELFMSEG4ATFBNBTS2EQ) context deadline exceeded
======LOGS======
26 14:18:38.467019958 (502h49m38.338102993s) [noise=true , address=145.79.227.40:4078 , piece_id=KYQVOIN6W54E7IMNEEO3QWGJ47NNZNQFGPQYMUQMQSXJ3E2IEJXA, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.467589142 (502h49m38.338672187s) [noise=true , address=77.90.31.14:4685 , piece_id=DGJ7OCZTT447YPGGJ4GAQFIUCJ2DGXGRUSOG4FH2WHJ34SVAZRQA, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.468070849 (502h49m38.339153884s) [noise=true , address=145.79.227.34:4933 , piece_id=DSKRIIY74OSDXAJWTRVXWKBCIOCNQJGH52ZIRJA5TYJAO57PTKPQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.468393304 (502h49m38.339476339s) [noise=true , address=193.149.164.143:4699 , piece_id=I5ETRK2KBL6BQBFQVQ2T3ZRBPPOEILP45ZR2QU2UJ4U5H5XKUHYQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.468696009 (502h49m38.339779045s) [noise=true , address=145.79.227.44:4197 , piece_id=CGDTN6BUBJKOTDCI3UGYYC2BKUPGEPIKZB63HFOAQTI7Q2WN4X6A, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.468954362 (502h49m38.340037397s) [noise=true , address=45.91.116.8:4063 , piece_id=LWEDMYQ7Z22TVDIXJAVOUU4CXOAGJ4NZUSPLJNGZASIWXEQJY3RQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.469329506 (502h49m38.340412541s) [noise=true , address=45.91.116.6:4988 , piece_id=H6BEM3BWWOHLBWAOKA4Q72THPYOFZECENPNNKZSAXL3TW4TDU6HQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.469594841 (502h49m38.340677876s) [noise=true , address=77.90.31.10:4528 , piece_id=U2YDYT4YEEWNEFQDQHSKGG4O6J5GJJRFZLL2DFBHGDBC7SITBGBQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.469889502 (502h49m38.340972537s) [noise=true , address=77.90.31.18:4826 , piece_id=4I6EKBOMAJ773LWPTUDSSJNJ2PHZSKN4O5M5VZG7KB7JLXENSUUQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.471501472 (502h49m38.342584507s) [noise=true , address=145.79.227.15:4894 , piece_id=GKTUQ7JEISCN2RRPQ7QGRLJJF5GRL2QSTI7VFTQARPNL45ZF5V6Q, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.472617888 (502h49m38.343700933s) [noise=true , address=45.91.116.9:4265 , piece_id=2N5MB2LVSOALPJNJXPIRYRLAE4V6WVV4TWGGD5ZIIR2WPSDUIAAA, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.473696321 (502h49m38.344779356s) [noise=true , address=77.90.31.21:4905 , piece_id=S2HIJM3SS3LVCQYJA47P3EINTRROGJOHZNA6DGEEGGEKER3O2V2A, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.475094033 (502h49m38.346177068s) [noise=true , address=77.90.31.11:4567 , piece_id=K3WQHFZWAZBM3FL7MKBJJEOGIVRXSFZEQELFMSEG4ATFBNBTS2EQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.476481184 (502h49m38.347564219s) [noise=true , address=145.79.227.49:4352 , piece_id=XN2XHVQFMCPY7MASXF4EARHF4JZUYZ2JSXLERJ5OYK2OOKLSOERA, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.477936085 (502h49m38.34901913s) [noise=true , address=45.91.116.6:4984 , piece_id=YPTT5PM5WNFWJ4ANVJDRHDWUVKVFTEHSWU3ZPBSPN3BAPHNS5Z4A, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.479119047 (502h49m38.350202082s) [noise=true , address=77.90.31.12:4604 , piece_id=62QMBJLUMECN5ZPIWH33MUK5V4FGNBNWH4TGNET3SZ7T7FP7J43Q, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:38.480145832 (502h49m38.351228867s) [noise=true , address=145.79.227.30:4782 , piece_id=XVJBWDJ7DVCMLXNBUILD3PAYC6QO3YPOCMPEL65NCQ3LO7G6SMBQ, seg_pos={0 0}, upload=441267]: Uploading piece...
Dec 26 14:18:39.935386623 (502h49m39.806469658s) [upload=441267]: Next returned nil segment
Dec 26 14:18:39.935404718 (502h49m39.806487783s) [upload=441267]: Waiting for error group managing segments...
Dec 26 14:18:40.359045076 (502h49m40.230128121s) [noise=true , address=145.79.227.49:4352 , piece_id=XN2XHVQFMCPY7MASXF4EARHF4JZUYZ2JSXLERJ5OYK2OOKLSOERA, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.361453582 (502h49m40.232536617s) [noise=true , address=145.79.227.15:4894 , piece_id=GKTUQ7JEISCN2RRPQ7QGRLJJF5GRL2QSTI7VFTQARPNL45ZF5V6Q, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.547237949 (502h49m40.418320994s) [noise=true , address=77.90.31.10:4528 , piece_id=U2YDYT4YEEWNEFQDQHSKGG4O6J5GJJRFZLL2DFBHGDBC7SITBGBQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.589525482 (502h49m40.460608517s) [noise=true , address=45.91.116.6:4988 , piece_id=H6BEM3BWWOHLBWAOKA4Q72THPYOFZECENPNNKZSAXL3TW4TDU6HQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.615065145 (502h49m40.48614818s) [noise=true , address=193.149.164.143:4699 , piece_id=I5ETRK2KBL6BQBFQVQ2T3ZRBPPOEILP45ZR2QU2UJ4U5H5XKUHYQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.636472638 (502h49m40.507555673s) [noise=true , address=77.90.31.14:4685 , piece_id=DGJ7OCZTT447YPGGJ4GAQFIUCJ2DGXGRUSOG4FH2WHJ34SVAZRQA, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.680324079 (502h49m40.551407114s) [noise=true , address=77.90.31.12:4604 , piece_id=62QMBJLUMECN5ZPIWH33MUK5V4FGNBNWH4TGNET3SZ7T7FP7J43Q, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:40.680630803 (502h49m40.551713838s) [noise=true , address=45.91.116.6:4984 , piece_id=YPTT5PM5WNFWJ4ANVJDRHDWUVKVFTEHSWU3ZPBSPN3BAPHNS5Z4A, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.014174410 (502h49m40.885257455s) [noise=true , address=77.90.31.11:4567 , piece_id=K3WQHFZWAZBM3FL7MKBJJEOGIVRXSFZEQELFMSEG4ATFBNBTS2EQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.274206518 (502h49m41.145289564s) [noise=true , address=145.79.227.44:4197 , piece_id=CGDTN6BUBJKOTDCI3UGYYC2BKUPGEPIKZB63HFOAQTI7Q2WN4X6A, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.284341067 (502h49m41.155424112s) [noise=true , address=145.79.227.30:4782 , piece_id=XVJBWDJ7DVCMLXNBUILD3PAYC6QO3YPOCMPEL65NCQ3LO7G6SMBQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.335517386 (502h49m41.206600431s) [noise=true , address=145.79.227.34:4933 , piece_id=DSKRIIY74OSDXAJWTRVXWKBCIOCNQJGH52ZIRJA5TYJAO57PTKPQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.427397291 (502h49m41.298480337s) [noise=true , address=77.90.31.21:4905 , piece_id=S2HIJM3SS3LVCQYJA47P3EINTRROGJOHZNA6DGEEGGEKER3O2V2A, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.760622433 (502h49m41.631705468s) [noise=true , address=145.79.227.40:4078 , piece_id=KYQVOIN6W54E7IMNEEO3QWGJ47NNZNQFGPQYMUQMQSXJ3E2IEJXA, seg_pos={0 0}, upload=441267]: Done uploading piece. err: <nil>
Dec 26 14:18:41.760645867 (502h49m41.631728922s) [seg_pos={0 0}, upload=441267]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:18:41.760900042 (502h49m41.631983077s) [noise=true , address=77.90.31.18:4826 , piece_id=4I6EKBOMAJ773LWPTUDSSJNJ2PHZSKN4O5M5VZG7KB7JLXENSUUQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6coC31yHW7RcRYPsoUqoYdgS7mrzUHWGMdH2Hma2QC7qe8JQ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:41.760910471 (502h49m41.631993537s) [noise=true , address=77.90.31.18:4826 , piece_id=4I6EKBOMAJ773LWPTUDSSJNJ2PHZSKN4O5M5VZG7KB7JLXENSUUQ, seg_pos={0 0}, upload=441267]: Long tail cancellation due to optimal threshold
Dec 26 14:18:41.760922795 (502h49m41.63200583s) [noise=true , address=45.91.116.9:4265 , piece_id=2N5MB2LVSOALPJNJXPIRYRLAE4V6WVV4TWGGD5ZIIR2WPSDUIAAA, seg_pos={0 0}, upload=441267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:41.760930459 (502h49m41.632013514s) [noise=true , address=45.91.116.9:4265 , piece_id=2N5MB2LVSOALPJNJXPIRYRLAE4V6WVV4TWGGD5ZIIR2WPSDUIAAA, seg_pos={0 0}, upload=441267]: Long tail cancellation due to optimal threshold
Dec 26 14:18:41.761160157 (502h49m41.632243202s) [noise=true , address=77.90.31.15:4702 , piece_id=6GM6LSN7GLD5YVGSSTCX3C5JVBI3JXDDPQUJH6REI6QREHA4LGVQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:41.761168542 (502h49m41.632251587s) [noise=true , address=77.90.31.15:4702 , piece_id=6GM6LSN7GLD5YVGSSTCX3C5JVBI3JXDDPQUJH6REI6QREHA4LGVQ, seg_pos={0 0}, upload=441267]: Long tail cancellation due to optimal threshold
Dec 26 14:18:41.761441643 (502h49m41.632524688s) [noise=true , address=45.91.116.8:4063 , piece_id=LWEDMYQ7Z22TVDIXJAVOUU4CXOAGJ4NZUSPLJNGZASIWXEQJY3RQ, seg_pos={0 0}, upload=441267]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1K5nU4YVCDnYSVvAWxbnWqB2pM4pTadgwqBWt4oeNpToz2X9xm): protocol: context canceled; context canceled
Dec 26 14:18:41.761450660 (502h49m41.632533695s) [noise=true , address=45.91.116.8:4063 , piece_id=LWEDMYQ7Z22TVDIXJAVOUU4CXOAGJ4NZUSPLJNGZASIWXEQJY3RQ, seg_pos={0 0}, upload=441267]: Long tail cancellation due to optimal threshold
Dec 26 14:18:41.761469406 (502h49m41.632552441s) [seg_pos={0 0}, upload=441267]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:18:41.761554327 (502h49m41.632637372s) [upload=441267]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:18:41.761560970 (502h49m41.632644005s) [upload=441267]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:18:42.068671854 (502h49m41.939754889s) [upload=441267]: Done uploading segments...
Dec 26 14:18:42.068693315 (502h49m41.939776351s) [upload=441267]: Upload finished. err: <nil>
Duration: 50.862450427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
16.057703029s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:20:25 UTC)
orbiter warning: context deadline exceeded
Duration: 16.057732585s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X..X.XXXXX.X..XX..XX...X...X...........X..X.X...........X..] duration:
6.235447007s trace:
https://jaeger.dev.storj.io/trace/3e839c16aad153a6
FAIL (2025-12-26 14:20:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q) context deadline exceeded
======LOGS======
14:18:47.356998944 (502h49m47.228081979s) [upload=441269]: Starting upload
Dec 26 14:18:47.357084037 (502h49m47.228167082s) [upload=441269]: Done starting upload
Dec 26 14:18:47.357123442 (502h49m47.228206487s) [upload=441269]: Uploading segments...
Dec 26 14:18:47.357439023 (502h49m47.228522068s) [upload=441269]: Got next segment. Inline: false
Dec 26 14:18:47.357479720 (502h49m47.228562755s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:18:47.357484669 (502h49m47.228567704s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:18:47.606177667 (502h49m47.477260702s) [seg_pos={0 0}, upload=441269]: Begin upload segment...
Dec 26 14:18:47.606230587 (502h49m47.477313622s) [seg_pos={0 0}, upload=441269]: Done begin upload segment.
Dec 26 14:18:47.606292986 (502h49m47.477376031s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606313534 (502h49m47.477396579s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.606988650 (502h49m47.478071686s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607483903 (502h49m47.478566938s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.607791419 (502h49m47.478874464s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608176933 (502h49m47.479259968s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608477235 (502h49m47.47956028s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.608795461 (502h49m47.479878496s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609114679 (502h49m47.480197724s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.609584413 (502h49m47.480667448s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610006628 (502h49m47.481089663s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610139139 (502h49m47.481222174s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610385598 (502h49m47.481468633s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.610784729 (502h49m47.481867764s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611152939 (502h49m47.482235974s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:47.611535077 (502h49m47.482618112s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Uploading piece...
Dec 26 14:18:49.321525799 (502h49m49.192608844s) [upload=441269]: Next returned nil segment
Dec 26 14:18:49.321539244 (502h49m49.192622279s) [upload=441269]: Waiting for error group managing segments...
Dec 26 14:18:49.600328349 (502h49m49.471411384s) [noise=true , address=145.79.227.19:4077 , piece_id=DJVQ3FWBN5ZDPEABH4MAKLMWXOVHD4HFFJYVKHMKVFRPLSUIVZSA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.607345371 (502h49m49.478428406s) [noise=true , address=145.79.227.5:4222 , piece_id=PY5I55FCSD7EXRMQSC3GC64RAXD3QYTHQWUZL5BVA5WIP6MU32IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.615918155 (502h49m49.48700122s) [noise=true , address=145.79.227.14:4837 , piece_id=PXXCQ7LRXLMR6FPNVVOG5HMVNZXU2CAZRTXVPH2UAASQZNQO2AWA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.620998108 (502h49m49.492081153s) [noise=true , address=145.79.227.36:4517 , piece_id=2GOUANWBI4KBXFOJUIGUHW2QLQ7TTBOFF2GSYUZ7MHOZZMGBTTHQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.621590276 (502h49m49.492673311s) [noise=true , address=145.79.227.23:4546 , piece_id=YRDLJHXWBTON3RUWAYP6WP7NSPQ4HBZEMKTXERG7JFS3FSJRIKZA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622164068 (502h49m49.493247103s) [noise=true , address=145.79.227.16:4910 , piece_id=VXG7XZZTCHG53YKJ6GR2QORTH2R3JXODUB7UQWEV2UDITZ5JXG4A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.622514555 (502h49m49.4935976s) [noise=true , address=145.79.227.18:4007 , piece_id=GK3Q4AX42MK2Q4NNWRHMDM2N2NR45RLAVO7TG2WZF5QQ6US7Q32Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.626668937 (502h49m49.497751982s) [noise=true , address=145.79.227.39:4015 , piece_id=BH3VNLCTM4X2USDY3NLL4EWM4RHZUWXJV6TGFVU2G47FYYX3Y7IQ, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.638758748 (502h49m49.509841793s) [noise=true , address=145.79.227.11:4732 , piece_id=S7ID7E4BB5T7TQLSVUI76OHMHCMPKLQDLLMBTEN25VYDJG4GC5UA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.641537970 (502h49m49.512621015s) [noise=true , address=145.79.227.29:4943 , piece_id=O5QWE3V6D2B5A77EWN7KSSUQFTLBHXIZETRONN3ZKD6ETYBBZRFA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.654182768 (502h49m49.525265803s) [noise=true , address=145.79.227.49:4339 , piece_id=B4YIB3VAYXQE4ZISU2JUNT3CZJDA64GGO6BTEDOV2N2ELSE3NZVA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.900555837 (502h49m49.771638882s) [noise=true , address=77.90.31.21:4878 , piece_id=W7RF5CDE57XWPXFIJXZOJTY7HGUEJQTYVOEJ4HFEDU2C6K2QE22Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919343812 (502h49m49.790426857s) [noise=true , address=77.90.31.14:4664 , piece_id=AEKVPJL3GW6MOPDPNIQN7YXLQMP2CHWG4XY6HOY7NWFMEVZJUE5Q, seg_pos={0 0}, upload=441269]: Done uploading piece. err: <nil>
Dec 26 14:18:49.919368009 (502h49m49.790451054s) [seg_pos={0 0}, upload=441269]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:18:49.919595642 (502h49m49.790678697s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59526->45.91.116.10:4081: use of closed network connection
Dec 26 14:18:49.919604949 (502h49m49.790688004s) [noise=true , address=45.91.116.10:4081 , piece_id=O5KXGMTCJZUOLMRCPPWVCPA32FZEJO4SY6EZZ757242LF7D2OL3A, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919779662 (502h49m49.790862707s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919795673 (502h49m49.790878718s) [noise=true , address=45.91.116.7:4012 , piece_id=L7FFBR4DIQZ4HBP42O2G5IZGCTYVLCKCP3TCCORHH5BR2TKNVNRA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919928807 (502h49m49.791011852s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 26 14:18:49.919934828 (502h49m49.791017863s) [noise=true , address=145.79.227.33:4870 , piece_id=AB7UKY3OK2U7IUZMXCMKOJ3P3X4VROGDQMMVAOZ7YX2LHCJ7IROA, seg_pos={0 0}, upload=441269]: Long tail cancellation due to optimal threshold
Dec 26 14:18:49.919959485 (502h49m49.79104252s) [seg_pos={0 0}, upload=441269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:18:49.920045398 (502h49m49.791128443s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:18:49.920050468 (502h49m49.791133503s) [upload=441269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:18:50.214541632 (502h49m50.085624687s) [upload=441269]: Done uploading segments...
Dec 26 14:18:50.214565568 (502h49m50.085648613s) [upload=441269]: Upload finished. err: <nil>
Duration: 6.235483186s
core
uplink
us-select-1
|