Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X.X.................................X.....X.........X...] duration:
18.875694645s trace:
https://jaeger.dev.storj.io/trace/79ab526980557de9
FAIL (2025-12-07 15:53:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.875722187s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.567581164s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:53:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.567611793s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.198034387s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:53:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.198067219s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X........X......X..............] duration:
6.109090763s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:53:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 6.109196355s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [XXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX.........] duration:
3.445880012s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:53:04 UTC)
orbiter warning: context deadline exceeded
Duration: 3.445905009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X.X.................................X.....X.........X..] duration:
18.169883604s trace:
https://jaeger.dev.storj.io/trace/79ab526980557de9
FAIL (2025-12-07 15:52:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.169966974s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.761627659s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:52:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.761652957s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
9.237811733s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:52:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.23784106s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [XXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX........] duration:
3.528517317s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:52:08 UTC)
orbiter warning: context deadline exceeded
Duration: 3.528546543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.424122286s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:52:08 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.424153474s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X........X......X.............] duration:
5.788227554s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:51:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 5.788253803s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.618061448s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:51:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.618090161s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X.X.................................X.....X.........X.] duration:
18.595568827s trace:
https://jaeger.dev.storj.io/trace/79ab526980557de9
FAIL (2025-12-07 15:51:43 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.595587143s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX.......] duration:
3.414998686s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:51:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.415039724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X......X.X.................................X.....X.........X] duration:
30.001173071s trace:
https://jaeger.dev.storj.io/trace/79ab526980557de9
FAIL (2025-12-07 15:51:25 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001200995s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.300016701s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:51:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.30004291s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.737548943s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:51:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.737639205s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X........X......X............] duration:
6.076561273s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:50:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 6.076589729s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
5.653260361s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:50:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.653289968s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.128386122s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:50:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.128415929s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X......X.X.................................X.....X.........] duration:
19.398419065s trace:
https://jaeger.dev.storj.io/trace/646b41ef1d2f671b
FAIL (2025-12-07 15:50:47 UTC)
orbiter warning: context deadline exceeded
Duration: 19.398448451s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
21 out of 60 is failed (threshold=6) [.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX......] duration:
3.401925003s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:50:42 UTC)
orbiter warning: context deadline exceeded
Duration: 3.401953817s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X........X......X...........] duration:
6.058793796s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:50:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 6.058825325s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.41997542s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:50:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.420014524s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X......X.X.................................X.....X........] duration:
19.071432s trace:
https://jaeger.dev.storj.io/trace/646b41ef1d2f671b
FAIL (2025-12-07 15:50:18 UTC)
orbiter warning: context deadline exceeded
Duration: 19.071463168s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX.....] duration:
3.392686888s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:50:11 UTC)
orbiter warning: context deadline exceeded
Duration: 3.392716043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
5.769677484s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:50:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.769710549s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X........X......X..........] duration:
5.663971212s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:49:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 5.664014926s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
13.550851688s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:49:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.550879652s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X......X.X.................................X.....X.......] duration:
23.565309506s trace:
https://jaeger.dev.storj.io/trace/646b41ef1d2f671b
FAIL (2025-12-07 15:49:57 UTC)
orbiter warning: context deadline exceeded
Duration: 23.565336949s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [.X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX....] duration:
3.516546965s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:49:53 UTC)
orbiter warning: context deadline exceeded
Duration: 3.516573805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.717864222s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:49:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.717922422s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.815620739s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:49:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.815658721s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X........X......X.........] duration:
6.518118394s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:49:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 6.518161406s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.196740803s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:49:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.196769056s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X......X.X.................................X.....X......] duration:
25.869022114s trace:
https://jaeger.dev.storj.io/trace/646b41ef1d2f671b
FAIL (2025-12-07 15:48:51 UTC)
orbiter warning: context deadline exceeded
Duration: 25.869052572s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
9.063822608s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:48:44 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.063847176s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
5.724833662s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:48:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.72486392s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
22 out of 60 is failed (threshold=6) [..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX...] duration:
3.498328485s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:48:39 UTC)
orbiter warning: context deadline exceeded
Duration: 3.498406604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.4098896s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:48:06 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.409919166s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X........X......X........] duration:
5.998581662s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:48:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 5.998620124s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.127141886s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:47:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 6.127168888s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X......X.X.................................X.....X.....] duration:
20.686301894s trace:
https://jaeger.dev.storj.io/trace/646b41ef1d2f671b
FAIL (2025-12-07 15:47:51 UTC)
orbiter warning: context deadline exceeded
Duration: 20.686329587s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [X..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX..] duration:
3.486930811s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:47:45 UTC)
orbiter warning: context deadline exceeded
Duration: 3.486965879s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X........X......X.......] duration:
7.532044632s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:47:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 7.532145364s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.745028416s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:47:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.745057751s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X........X......X......] duration:
8.14015052s trace:
https://jaeger.dev.storj.io/trace/76649d75ca4b1745
FAIL (2025-12-07 15:47:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ) context deadline exceeded
======LOGS======
Dec 7 15:42:52.075141337 (48h13m51.946224372s) [upload=42393]: Starting upload
Dec 7 15:42:52.075200600 (48h13m51.946283635s) [upload=42393]: Done starting upload
Dec 7 15:42:52.075355735 (48h13m51.94643877s) [upload=42393]: Uploading segments...
Dec 7 15:42:52.075629006 (48h13m51.946712051s) [upload=42393]: Got next segment. Inline: false
Dec 7 15:42:52.075673029 (48h13m51.946756074s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 7 15:42:52.075679713 (48h13m51.946762758s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 7 15:42:52.646860140 (48h13m52.517943185s) [seg_pos={0 0}, upload=42393]: Begin upload segment...
Dec 7 15:42:52.646921907 (48h13m52.518004953s) [seg_pos={0 0}, upload=42393]: Done begin upload segment.
Dec 7 15:42:52.646965200 (48h13m52.518048235s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.646982794 (48h13m52.518065829s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.647887376 (48h13m52.518970411s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648182538 (48h13m52.519265583s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648286496 (48h13m52.519369541s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648641683 (48h13m52.519724718s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.648678743 (48h13m52.519761778s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649046653 (48h13m52.520129698s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649472815 (48h13m52.52055586s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.649940345 (48h13m52.521023471s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650032391 (48h13m52.521115436s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650436259 (48h13m52.521519304s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.650739918 (48h13m52.521822963s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651153676 (48h13m52.522236721s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651585758 (48h13m52.522668803s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:52.651989718 (48h13m52.523072763s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Uploading piece...
Dec 7 15:42:54.440983928 (48h13m54.312066973s) [upload=42393]: Next returned nil segment
Dec 7 15:42:54.440996322 (48h13m54.312079367s) [upload=42393]: Waiting for error group managing segments...
Dec 7 15:42:54.795710406 (48h13m54.666793451s) [noise=true , address=77.90.31.11:4555 , piece_id=XF6573Y5K7U5EMAG4UXJVKGKZ4CV6B7BTKX72A7VQ5FKHSAUB4UA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.796230045 (48h13m54.66731309s) [noise=true , address=145.79.227.26:4655 , piece_id=IZUHGJ252JRTAMDDJN5UOTK7HUI5F4MY7ATLEP3SY6ZKSM6DGWWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804219258 (48h13m54.675302303s) [noise=true , address=145.79.227.42:4121 , piece_id=ZQADIICSJTIFZOZ5PZ22OP2SMEZ253TAGRZ4CKDVJD2WDRWQMKYQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.804402307 (48h13m54.675485352s) [noise=true , address=145.79.227.34:4906 , piece_id=HJNCYKVGL275P4GMPXSOHIGPTHAKHGTKQIHSUGRCEB26X2N3PCWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.820310880 (48h13m54.691393915s) [noise=true , address=145.79.227.43:4158 , piece_id=QHESFDBIBJ4IIN3DJEWNBLNMMGZFCMHRA3OOFWH36XHQCDID4A7A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.824943301 (48h13m54.696026346s) [noise=true , address=145.79.227.25:4619 , piece_id=MKXJFMPPVXXFZS6KOBKUS4MOG6TSQDBSUBABPJBH36TFPFVLRKXQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.828361219 (48h13m54.699444264s) [noise=true , address=145.79.227.38:4977 , piece_id=EPDALOSUS7Q7G53GAYDEIIT6QPLV3UFW4XTHY5XDZJ47PBYOBZMQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.857739461 (48h13m54.728822506s) [noise=true , address=145.79.227.39:4015 , piece_id=T6RK3GO6XPFGNMYMTXGK6DNGFOPJ4CVBW5QR6WCGE3OIY5B67IOA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:54.865338421 (48h13m54.736421456s) [noise=true , address=145.79.227.28:4727 , piece_id=JT3YPZTIFYO7FBRKLKPP2XUGERV7A5VHZVEXAZZWHODNOH2XT5JQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.108668117 (48h13m54.979751152s) [noise=true , address=145.79.227.20:4438 , piece_id=OYPJC44HAY6FMDVWBWFRVL326KPOBLX2BM3PUC54JNR7QZ7T6CDQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.204450035 (48h13m55.07553308s) [noise=true , address=145.79.227.21:4473 , piece_id=UHNFNYBLGGH2YR636ZO6LSDFTU6VIUWKRKXUGZLLGRQ3U45LFSOQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.495245134 (48h13m55.366328179s) [noise=true , address=145.79.227.24:4582 , piece_id=ZUJQHKNTEFKDLHEOQHKRH4HGYV7OMMX3QG3GAPCPSJQRUS3OEA5A, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528102311 (48h13m55.399185356s) [noise=true , address=145.79.227.23:4545 , piece_id=FGWEVBVVC454K3RL7HCIBSFN62PD7C7XMIDVJ5GBG6LSFTC3YBCQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: <nil>
Dec 7 15:42:55.528123241 (48h13m55.399206287s) [seg_pos={0 0}, upload=42393]: Segment reached optimal threshold of 13 pieces.
Dec 7 15:42:55.528359992 (48h13m55.399443028s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528366525 (48h13m55.39944956s) [noise=true , address=145.79.227.38:4978 , piece_id=MN2YFDS4FVAUI2LQ4EUGNXIGHVXU3KK3OAHUVY2NC4IZAWUU34BQ, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528488256 (48h13m55.399571301s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38404->145.79.227.40:4049: use of closed network connection
Dec 7 15:42:55.528505860 (48h13m55.399588895s) [noise=true , address=145.79.227.40:4049 , piece_id=YCQCAZWS4KJ3NETGDL2N4MKE4LINWZ2TNIG5GKUUZYK4XRMV4CWA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528539404 (48h13m55.399622449s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF
Dec 7 15:42:55.528548120 (48h13m55.399631155s) [noise=true , address=145.79.227.41:4087 , piece_id=Z6VYEJS7GNUDAVQKT7N5LAZ3FBR7MMZJMVLBLO3S5UHBVIRWS5VA, seg_pos={0 0}, upload=42393]: Long tail cancellation due to optimal threshold
Dec 7 15:42:55.528619466 (48h13m55.399702501s) [seg_pos={0 0}, upload=42393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 7 15:42:55.528697785 (48h13m55.39978083s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 7 15:42:55.528718635 (48h13m55.39980168s) [upload=42393]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 7 15:42:55.843997748 (48h13m55.715080783s) [upload=42393]: Done uploading segments...
Dec 7 15:42:55.844020370 (48h13m55.715103415s) [upload=42393]: Upload finished. err: <nil>
Duration: 8.140175737s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
5.347562637s trace:
https://jaeger.dev.storj.io/trace/30a119e66e6df16c
FAIL (2025-12-07 15:46:51 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.347685701s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.244396647s trace:
https://jaeger.dev.storj.io/trace/6aa77812f5688c1c
FAIL (2025-12-07 15:46:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130342060 (47h58m9.001425105s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.130939638 (47h58m9.002022683s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.134252346 (47h58m9.005335381s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.140370596 (47h58m9.011453631s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.143401798 (47h58m9.014484833s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.148359669 (47h58m9.019442704s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.151439022 (47h58m9.022522057s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.154068229 (47h58m9.025151264s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.163005166 (47h58m9.034088211s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.164528286 (47h58m9.035611322s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.165982556 (47h58m9.037065591s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.166671929 (47h58m9.037754964s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.168120026 (47h58m9.039203061s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Uploading piece...
Dec 7 15:27:09.953467338 (47h58m9.824550373s) [upload=42166]: Next returned nil segment
Dec 7 15:27:09.953481746 (47h58m9.824564781s) [upload=42166]: Waiting for error group managing segments...
Dec 7 15:27:10.350054998 (47h58m10.221138043s) [noise=true , address=185.167.204.206:4064 , piece_id=R5STPZC34OPJYLPWTSKXGDTY4KOHDTTZJEKLWPC3AT7XKV6EQK5Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350209161 (47h58m10.221292196s) [noise=true , address=185.167.204.206:4041 , piece_id=IZRUG2UEYE4FIYTBOVLW5DUNB3DY54J3QJRFI72LYYTZR7TTSQSQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350474396 (47h58m10.221557441s) [noise=true , address=185.167.204.206:4056 , piece_id=NPX4AZ3AC6ELYWLDPCPVL6QCW5GGWZAJN4O7IXVCT5NY6UXRC6SQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.350809705 (47h58m10.22189274s) [noise=true , address=185.167.204.206:4071 , piece_id=JLKZC3DAVXDHPVZAYCPCFWCS2NPPCK52BCLPUGGZHNEMUPQV45LQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.351833624 (47h58m10.222916659s) [noise=true , address=185.167.204.206:4037 , piece_id=N7UNV4PCLGWG7NUKYKYN4KWJBW7TLX6F6WHIYQXTH2JWB7L7EO3Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.352044826 (47h58m10.223127861s) [noise=true , address=185.167.204.205:4033 , piece_id=AMP7ZR43NFBFOZBN7TUKKK2BM36JZH3H42LUCTYUURCOHDEM2A6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354075954 (47h58m10.225158999s) [noise=true , address=185.167.204.205:4011 , piece_id=UUMS33NMBLUKXLHRA7VWO7ZMRJQGV4LTWFRJO2WNCAI6E6AO6RKA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354562991 (47h58m10.225646026s) [noise=true , address=185.167.204.205:4005 , piece_id=QTITHKWY5QQTTOQRLZQHJS26JP64SXMPOVIJNDV5Q6DLKCTU7PDQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.354642433 (47h58m10.225725478s) [noise=true , address=185.167.204.205:4009 , piece_id=NP4VOZNL6SXY5OTCQHQYHCHZXP77HUJFGNH3X6OA6MMWUJRSWO3A, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.361143913 (47h58m10.232226948s) [noise=true , address=185.167.204.206:4043 , piece_id=CL4FR7ZZGP5CPTABC6FLXCEBFNPRSFKXCMTOOLZ7L66N4EIUHIQQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.367126485 (47h58m10.23820953s) [noise=true , address=185.167.204.206:4052 , piece_id=V6LGIM4Z2G5SA7CD4EXTFDLY4PLFZBOZ4UURLFIJWIS6DTGQXVEA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.370543752 (47h58m10.241626787s) [noise=true , address=185.167.204.205:4013 , piece_id=4XSNAA6BUMB4RVOULYTRXGAQEUSYU7AG6PX45LM57UBGEQ6COX7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.373445157 (47h58m10.244528202s) [noise=true , address=185.167.204.205:4015 , piece_id=CH3A7LP4RWIJUO5RW3SPAZD2NR3TOJRILIW3IB76WMOTAY4DZ7SA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.378621894 (47h58m10.249704939s) [noise=true , address=185.167.204.205:4021 , piece_id=BJIJNBEYDCGDIXPI3WYEGXQBZZY65B2Y2JX7FWQG6SAOPCVXXVMQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.382356677 (47h58m10.253439712s) [noise=true , address=185.167.204.205:4031 , piece_id=UHZJVGM2AAK7QP3XQ6UWZ7NCU2LA5MG5WGF2ICO3VLRW3Y2YIDEQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.388414452 (47h58m10.259497497s) [noise=true , address=185.167.204.205:4020 , piece_id=EHDW3VHPIIDRAJIRQLVJAHD5RACGQLVNSKCF4JGF5TSYD3DVZARA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.390471939 (47h58m10.261554974s) [noise=true , address=185.167.204.206:4055 , piece_id=ZDTBPHVAWLIAM2ABKFDK7TRTCV5WVKWJI5SW4PSDPVZHQ2GCYUWA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920146511 (47h58m10.791229556s) [noise=true , address=81.171.14.70:4038 , piece_id=VH5D3TJ3ZOUSLZ5BGFTT6D3IDPMBFNVC5TZ6E5SEU4MDKINED3XQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.920343136 (47h58m10.791426171s) [noise=true , address=81.171.14.70:4035 , piece_id=BTYN6MPRPSQW772VSFP27H2KSC2WEGHN3T3DS7QBB6NFOKG67RTA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.924686387 (47h58m10.795769442s) [noise=true , address=81.171.14.70:4043 , piece_id=OXROZDIUIIMCCYTANLIRMHUDBSXHFHGI4CK34DUFEIC5ELEX4B6Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.926977780 (47h58m10.798060825s) [noise=true , address=81.171.14.70:4044 , piece_id=4CVUK4EK66VLJKLYHMFEA7LG2YZIWHCZMXLZ4NZA7YGMTLUC2RFQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.928636548 (47h58m10.799719583s) [noise=true , address=81.171.14.70:4031 , piece_id=DVOLJCA4OJDGBUICGCW3TPBWFM2LRPSDU3PTMXT6HJIBUBRDAPFA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:10.951702800 (47h58m10.822785836s) [noise=true , address=81.171.14.70:4039 , piece_id=G3DQ5UQGXUMWQL4HPK2OXNA5E6LLYLXGP6B5T7Q6PCRI53FWFGDA, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.399202848 (47h58m11.270285883s) [noise=true , address=81.171.14.70:4030 , piece_id=LWBEFXIDNKFWHJVXMIUMIPMGAGUUG6ETE6ZQV3NUVSUL766YHSRQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:27:11.478044017 (47h58m11.349127062s) [noise=true , address=81.171.14.70:4028 , piece_id=FQZQGAFLPDZUGYQF6V7LZRBYNHN4FUI74I2BRMHN35NVNN6KM2HQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: <nil>
Dec 7 15:28:08.469008783 (47h59m8.340091828s) [noise=true , address=89.149.201.147:4016 , piece_id=HLLLLW2QHZR2CMEHWTQTS52AOBWRMUHVJTAB4MLOVLZHREG5VZ7Q, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5, address:89.149.201.147:4016): protocol: read tcp 188.166.225.170:56386->89.149.201.147:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 7 15:28:08.469094447 (47h59m8.340177492s) [noise=true , address=89.149.201.147:4009 , piece_id=KSKTR2MSKNOEGTEHDC3I373JLRRG43SA6COH2OWVPJALXVWNXAAQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F, address:89.149.201.147:4009): protocol: expected piece hash; context deadline exceeded; EOF
Dec 7 15:28:08.469319024 (47h59m8.340402069s) [noise=true , address=89.149.201.147:4024 , piece_id=XVGAQ7NI4ORBFRTUEN5N2M2PGW34UX2QDMPTK2S5MMH6455CVEBQ, seg_pos={0 0}, upload=42166]: Done uploading piece. err: ecclient: upload failed (node:12r2sAmG1Xa83zRjY96g1Ro6dfDUHvKkqdCtFYoeviKcUJsAMBT, address:89.149.201.147:4024): protocol: context deadline exceeded; context deadline exceeded
Dec 7 15:28:08.469355052 (47h59m8.340438108s) [seg_pos={0 0}, upload=42166]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Dec 7 15:28:08.637507067 (47h59m8.508590103s) [upload=42166]: Done uploading segments...
Dec 7 15:28:08.637550680 (47h59m8.508633725s) [upload=42166]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.244420202s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [XX..X.XXXXXX..X.X....X....X.X.........XX..X.....XXX..X.X.XX.] duration:
3.514361452s trace:
https://jaeger.dev.storj.io/trace/6c4eaa740f2240e2
FAIL (2025-12-07 15:46:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.514387361s
core
uplink
us-select-1
us1-placement-20
|