Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.888733099s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:32:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.888758147s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X.........X.........................] duration: 3.432085175s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:32:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.432114731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.090438478s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:31:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.090471811s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.149463356s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:31:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.149493814s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X.........X........................] duration: 4.429432584s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:31:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.429459036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.749787075s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:30:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.749826581s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X.........X.......................] duration: 3.788811524s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:30:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.788839389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.........X......................] duration: 3.425530054s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:29:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425558427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.261299944s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:29:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.261332044s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.........X.....................] duration: 3.392756975s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:29:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392788434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.........X....................] duration: 3.475717203s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:28:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476069375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.047817654s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:28:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.047849295s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.........X...................] duration: 3.510424128s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:28:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510455797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X.........X..................] duration: 3.53826894s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:27:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538552711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.........X.................] duration: 3.433268017s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:27:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433297423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.946311108s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:27:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.946343711s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.........X................] duration: 3.427557996s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:27:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42759633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.431400501s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.431431299s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X.........X...............] duration: 3.425061563s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:26:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.42509655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.294940999s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:26:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.294972319s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X.........X..............] duration: 3.461045613s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:26:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.461079618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.3277682s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:25:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.327806594s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X.........X.............] duration: 3.492330414s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:25:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49236406s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X.........X............] duration: 3.471011301s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:24:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471081545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.056426435s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:24:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.056453525s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X.........X...........] duration: 3.548657122s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:23:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54869259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.117459803s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:23:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.117492185s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X......................................X.........X..........] duration: 3.510854277s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:22:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510888022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.183894295s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:22:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.183924081s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X......................................X.........X.........] duration: 3.445429284s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:22:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445461896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X......................................X.........X........] duration: 3.436018577s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:21:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.436053032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.923534609s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:20:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.923564646s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X......................................X.........X.......] duration: 3.427344628s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:20:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427371239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.807750231s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:20:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.807784286s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X......................................X.........X......] duration: 4.371277102s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:19:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.371319484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.352777633s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:19:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.352814504s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X......................................X.........X.....] duration: 3.478530437s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:18:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478560103s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X......................................X.........X....] duration: 3.502005728s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:18:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502117421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.171242354s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:18:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.171274035s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.890242142s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:17:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.890273843s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X......................................X.........X...] duration: 3.428701061s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:17:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428728364s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X......................................X.........X..] duration: 3.439381191s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:17:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439409146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.99038525s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:16:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 5.990410719s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X......................................X.........X.] duration: 3.50588247s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:16:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50657047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.264011827s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:15:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.264029541s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X......................................X.........X] duration: 5.001237196s trace: https://jaeger.dev.storj.io/trace/7268fb85428c9e52 FAIL (2025-11-06 14:15:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001263267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.301263466s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:15:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.301285598s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.427258021s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:15:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.427289861s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X......................................X.........] duration: 3.442932329s trace: https://jaeger.dev.storj.io/trace/57f49e58cb7314cb FAIL (2025-11-06 14:14:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442967204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.162782425s trace: https://jaeger.dev.storj.io/trace/6f7930158ec18dea FAIL (2025-11-06 14:14:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 25) ======LOGS====== dress=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967491189 (170h16m41.663339932s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967873697 (170h16m41.66372245s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.967951827 (170h16m41.66380057s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968261997 (170h16m41.66411075s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968432701 (170h16m41.664281454s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.968895694 (170h16m41.664744447s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969397699 (170h16m41.665246452s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.969914393 (170h16m41.665763146s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.970231888 (170h16m41.666080641s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.971781058 (170h16m41.667629811s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:09.972571684 (170h16m41.668420427s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Uploading piece... Nov 6 13:57:10.827880794 (170h16m42.523729547s) [upload=150053]: Next returned nil segment Nov 6 13:57:10.827905561 (170h16m42.523754314s) [upload=150053]: Waiting for error group managing segments... Nov 6 13:57:11.231640032 (170h16m42.927488785s) [noise=true , address=81.171.14.70:4042 , piece_id=YCONCKCIBPX7KMMAR5MEDBIY5CR7ZW26AWUHLB4EWRFMEDQYLD7Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.256442090 (170h16m42.952290843s) [noise=true , address=81.171.14.70:4043 , piece_id=SB4C6URXGDMVMMEL4KEKILGUEWQKQL5OOQOTPIVNM64S4EZTJJWQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.257443336 (170h16m42.953292089s) [noise=true , address=81.171.14.70:4041 , piece_id=RDIDN2YMS4FZYBKQPPQYZKEEPDMVGSTKCTHGPIMKKM7IPFV5YDBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258387294 (170h16m42.954236047s) [noise=true , address=89.149.201.147:4015 , piece_id=QIRYWFPQR5NTPI4O4MZAFBFSDJ64X6QNGFSXGUIGXES5MLPYKWBQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.258894860 (170h16m42.954743603s) [noise=true , address=89.149.201.147:4005 , piece_id=NUBPG2CZKYOFKKEDATQNMIIEMRJLKCV7BXON74JJTKOZ4O3RLXYA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.261496525 (170h16m42.957345278s) [noise=true , address=89.149.201.147:4020 , piece_id=JVF7STR6OLUCTUSYV6CXFVIKVATTOJHEAEJIQ4GQ6PYBSY5JIWHA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.272583977 (170h16m42.968432741s) [noise=true , address=89.149.201.147:4017 , piece_id=NM7LFEV6SHZVSEVMYISA6HXJ35MNPNHXW73OLALOUHZHNSFUD25Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.278466539 (170h16m42.974315282s) [noise=true , address=81.171.14.70:4034 , piece_id=UY72SXBSXJEDYGSBDCR2EQQU7ONF4JECXUYCHNFFVLEBPB7EP3SA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.306805412 (170h16m43.002654165s) [noise=true , address=89.149.201.147:4009 , piece_id=DMSFQWMG6LGGW5RIXHNAFSMTB4NQOLGRDOCQNM2P7HQ7PB7XIOAQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.307888204 (170h16m43.003736967s) [noise=true , address=89.149.201.147:4010 , piece_id=AXUNULAVIM67QI4RV52PJCWXGFHS2WFDBX4R57MSRRTC4NPN5JLQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.343739372 (170h16m43.039588125s) [noise=true , address=89.149.201.147:4012 , piece_id=KLXS663OUTFUL2FPSTCJPP5N5BGDC7O5JCKSMEP4S633GNB6HAJA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.716425841 (170h16m43.412274594s) [noise=true , address=89.149.201.147:4024 , piece_id=BRGY7OUUISINCEPND2NESFP3XZ7FLG4DFW3RNDYXJUHXSKGGZ4DQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.753901162 (170h16m43.449749925s) [noise=true , address=89.149.201.147:4023 , piece_id=4JE3VN4QUOJOJQ36ELHXGSTYM6RVSAWJVGVCFMEYVYU22YZ2YXTQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.805773276 (170h16m43.501622029s) [noise=true , address=89.149.201.147:4011 , piece_id=KFKJYMKWZSIYQ46NMLBKY24BYPDRWCRG7TAJEIW55JIHR6T5R3KA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:11.815857448 (170h16m43.511706191s) [noise=true , address=89.149.201.147:4013 , piece_id=HE7XBI3JR45JCMXGUPGCIWEJANBU7BIFF3JVDARYIZEZUBDMVXXA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.467950153 (170h16m44.163798906s) [noise=true , address=185.167.204.205:4013 , piece_id=KBYH7LUODR6RSEKOCWZN46PATU5OSKSQF2AUTN45KZYZOXYRZC3Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.481087159 (170h16m44.176935912s) [noise=true , address=185.167.204.205:4021 , piece_id=OLV3IKKXTCXKOUSZF2NTTDQXA46SF4TATXICAEWFJQGSMKLNB2YQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.485759787 (170h16m44.18160854s) [noise=true , address=185.167.204.206:4056 , piece_id=SUP37NDMXS4CDS5Z7XSPABOTYHWCGVVJCPRTSTXZ4S4KQN73I7HQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.491917152 (170h16m44.187765905s) [noise=true , address=185.167.204.205:4014 , piece_id=D7VDIU7AKGGT2JVZXBSMXM2DKG2ILJND5SGE2EHHJDKL7QVZQJZA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.496242337 (170h16m44.19209109s) [noise=true , address=185.167.204.205:4010 , piece_id=VQN66GUL5K3MTZKK4ZD2ZEF47BFXFC6KTGYNL3W4YYY3BANPZRJQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.503381952 (170h16m44.199230705s) [noise=true , address=185.167.204.205:4035 , piece_id=PHCM37Y4ERDT3LXI2OIJCXDY6TC4ZTAHEOJKJNERQD7XY2I2VLCQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.508231517 (170h16m44.204080281s) [noise=true , address=185.167.204.206:4058 , piece_id=4VHBMCJBKLMPI2ARQFFPS5CSSMY6RYVB24ERFARVGDWR3NTJR4IA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.552927117 (170h16m44.24877587s) [noise=true , address=185.167.204.205:4007 , piece_id=J5ZPRWBIY3ATFERVVCEBLRYI6MXEURNZA6KAVCVX4Y5FEGY6H4IQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.554453173 (170h16m44.250301926s) [noise=true , address=185.167.204.206:4040 , piece_id=UNZQTD5MKIWKVAH7BMCIQD527PIWTGOYEYXTVYIZVVW2PVI7XUKQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:57:12.555070939 (170h16m44.250919692s) [noise=true , address=185.167.204.205:4034 , piece_id=M3AEVYF5GSIJIBHDF5W22PEILMREFIP76J6MQ6L25VZQJWFKEGXQ, seg_pos={0 0}, upload=150053]: Done uploading piece. err: <nil> Nov 6 13:58:09.307307484 (170h17m41.003156237s) [noise=true , address=185.167.204.205:4026 , piece_id=LUUWBTTDXT46HJOYAXPUQVOASM7E34JFB5ATDZIXQ7VQSVFNUSCA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1A9VrfmePrZ6AiT4SgVgEEahfBc5u5e2LnQm86gEpEkQgqB2e4, address:185.167.204.205:4026): protocol: write tcp 188.166.225.170:34426->185.167.204.205:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307473691 (170h17m41.003322464s) [noise=true , address=185.167.204.206:4072 , piece_id=X62JXCQKFGXZNUFIAJ6TQT2NQO5CEJFRKE5TRJ7XCUZVMSB2YDBA, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: write tcp 188.166.225.170:34834->185.167.204.206:4072: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307485523 (170h17m41.003334266s) [noise=true , address=185.167.204.206:4066 , piece_id=ZKA5JBI26A332CUNON7FMGEERZWWZ2A22SRAV254HE2B23F25K2Q, seg_pos={0 0}, upload=150053]: Done uploading piece. err: ecclient: upload failed (node:1ns5eXmL9FCPxzevdEpYg6HZEjHnE3ZRB6JRhY51hMMW2mEmUg, address:185.167.204.206:4066): protocol: write tcp 188.166.225.170:38838->185.167.204.206:4066: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 6 13:58:09.307508226 (170h17m41.003356979s) [seg_pos={0 0}, upload=150053]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil> Nov 6 13:58:09.482426726 (170h17m41.178275479s) [upload=150053]: Done uploading segments... Nov 6 13:58:09.482474207 (170h17m41.17832296s) [upload=150053]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25) Duration: 6.1628115s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)